2024-04-26T22:32:15.545 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T22:32:15.545 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} --owner scheduled_rishabh@teuthology --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --verbose -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/orig.config.yaml 2024-04-26T22:32:15.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T22:32:15.693 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} email: null first_in_suite: false job_id: '7675272' kernel: client: sha1: distro flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnCjic0Oq+Ct/r3wvi53lH5O+2bqC9vZ5P/jeGDjOmhzU/05hM2CJTwJeaz+OmcJe6HH95VqMjRNe5+9eCl9sQ= smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOb4nixO2kZ88v5vmrAbE4znWB8mC3hnrVxOJkfOERbEVQSLMGSKs4w/KFSA4dT7cA/tLdt9w+Q3OI42nt6kvns= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxiJxGhunQFdo4YSRlfWv+ih6Pcpphd5HwDNkvzsTsxwgDAljeZ77XUxMCiM2XarVuznIgkvtB/C9S/NjdGwtI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T22:32:15.693 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T22:32:15.694 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T22:32:15.694 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T22:32:15.695 INFO:teuthology.task.internal:Checking packages... 2024-04-26T22:32:15.718 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T22:32:15.719 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T22:32:15.719 INFO:teuthology.packaging:ref: None 2024-04-26T22:32:15.719 INFO:teuthology.packaging:tag: None 2024-04-26T22:32:15.719 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T22:32:15.719 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:32:15.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T22:32:15.940 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T22:32:15.941 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T22:32:15.950 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T22:32:15.950 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T22:32:16.034 INFO:teuthology.task.internal:Saving configuration 2024-04-26T22:32:16.053 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T22:32:16.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T22:32:16.086 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:27:19.644092', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnCjic0Oq+Ct/r3wvi53lH5O+2bqC9vZ5P/jeGDjOmhzU/05hM2CJTwJeaz+OmcJe6HH95VqMjRNe5+9eCl9sQ='} 2024-04-26T22:32:16.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:27:19.646609', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOb4nixO2kZ88v5vmrAbE4znWB8mC3hnrVxOJkfOERbEVQSLMGSKs4w/KFSA4dT7cA/tLdt9w+Q3OI42nt6kvns='} 2024-04-26T22:32:16.127 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:27:19.645445', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxiJxGhunQFdo4YSRlfWv+ih6Pcpphd5HwDNkvzsTsxwgDAljeZ77XUxMCiM2XarVuznIgkvtB/C9S/NjdGwtI='} 2024-04-26T22:32:16.127 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T22:32:16.135 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T22:32:16.135 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T22:32:16.135 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T22:32:16.135 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T22:32:16.297 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc5e8aa8160>, signals=[15]) 2024-04-26T22:32:16.298 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T22:32:16.305 INFO:teuthology.task.internal:Opening connections... 2024-04-26T22:32:16.305 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T22:32:16.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:32:16.385 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-26T22:32:16.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:32:16.463 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T22:32:16.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:32:16.535 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T22:32:16.543 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-04-26T22:32:16.562 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-04-26T22:32:16.562 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T22:32:16.620 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-04-26T22:32:16.621 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-04-26T22:32:16.621 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T22:32:16.621 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-04-26T22:32:16.621 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T22:32:16.621 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T22:32:16.621 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T22:32:16.621 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-04-26T22:32:16.642 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-26T22:32:16.661 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-26T22:32:16.661 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-26T22:32:16.719 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T22:32:16.720 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-26T22:32:16.720 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T22:32:16.720 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T22:32:16.720 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T22:32:16.720 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-26T22:32:16.742 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-26T22:32:16.760 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-26T22:32:16.761 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T22:32:16.818 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-26T22:32:16.819 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-26T22:32:16.819 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T22:32:16.819 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-26T22:32:16.819 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T22:32:16.819 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T22:32:16.819 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T22:32:16.819 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-26T22:32:16.843 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T22:32:16.853 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T22:32:16.901 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T22:32:16.901 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-04-26T22:32:16.904 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-26T22:32:16.906 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-26T22:32:16.922 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T22:32:16.931 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T22:32:16.931 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-04-26T22:32:16.962 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-26T22:32:16.965 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-26T22:32:16.989 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T22:32:17.063 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T22:32:17.077 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T22:32:17.077 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-26T22:32:17.077 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-26T22:32:17.078 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T22:32:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:32:17.096 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-04-26T22:32:17.153 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-437.el9.x86_64 2024-04-26T22:32:17.154 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-437.el9.x86_64 2024-04-26T22:32:17.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-04-26T22:32:20.271 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-04-26T22:32:22.357 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-26T22:32:26.689 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-26T22:32:28.298 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-26T22:32:29.216 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-04-26T22:32:34.449 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-04-26T22:32:36.515 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-26T22:32:36.515 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-26T22:32:36.613 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-04-26T22:32:36.614 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-04-26T22:32:36.614 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-04-26T22:32:36.728 DEBUG:teuthology.orchestra.run.smithi001:> echo no | sudo yum reinstall kernel || true 2024-04-26T22:32:37.291 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Apr 2024 10:32:34 PM UTC. 2024-04-26T22:32:37.414 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-04-26T22:32:37.430 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-04-26T22:32:37.430 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-04-26T22:32:37.430 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-04-26T22:32:37.430 INFO:teuthology.orchestra.run.smithi001.stdout:Reinstalling: 2024-04-26T22:32:37.430 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T22:32:37.430 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:32:37.431 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-04-26T22:32:37.431 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-04-26T22:32:37.431 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:32:37.431 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 255 k 2024-04-26T22:32:37.431 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 0 2024-04-26T22:32:37.431 INFO:teuthology.orchestra.run.smithi001.stderr:Operation aborted. 2024-04-26T22:32:37.501 INFO:teuthology.orchestra.run.smithi001.stdout:Is this ok [y/N]: 2024-04-26T22:32:37.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum reinstall -y kernel || true 2024-04-26T22:32:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:04 ago on Fri 26 Apr 2024 10:32:34 PM UTC. 2024-04-26T22:32:38.138 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:Reinstalling: 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 255 k 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 0 2024-04-26T22:32:38.139 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-04-26T22:32:38.793 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-26T22:32:38.793 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:32:38.793 INFO:teuthology.orchestra.run.smithi001.stdout:Total 389 kB/s | 255 kB 00:00 2024-04-26T22:32:38.813 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-04-26T22:32:38.994 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-04-26T22:32:38.995 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-04-26T22:32:38.999 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-04-26T22:32:39.000 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-04-26T22:32:39.119 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-04-26T22:32:39.186 INFO:teuthology.orchestra.run.smithi001.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T22:32:39.186 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T22:32:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T22:32:39.892 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T22:32:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T22:32:40.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:32:40.704 INFO:teuthology.orchestra.run.smithi001.stdout:Reinstalled: 2024-04-26T22:32:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-26T22:32:40.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:32:40.704 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-04-26T22:32:40.866 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T22:32:40.915 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-26T22:32:40.915 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-26T22:32:40.916 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-26T22:32:40.916 DEBUG:teuthology.parallel:result is None 2024-04-26T22:32:40.916 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T22:32:40.923 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T22:32:40.923 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T22:32:40.925 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T22:32:40.928 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T22:32:40.945 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T22:32:41.000 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T22:32:41.035 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T22:32:41.035 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T22:32:41.038 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T22:32:41.040 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T22:32:41.077 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T22:32:41.083 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T22:32:41.084 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T22:32:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:32:41.116 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T22:32:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:32:41.134 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T22:32:41.136 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T22:32:41.159 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T22:32:41.189 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:32:41.202 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:32:41.206 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:32:41.208 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:32:41.225 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:32:41.240 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:32:41.242 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T22:32:41.249 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T22:32:41.249 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T22:32:41.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T22:32:41.254 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T22:32:41.308 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T22:32:41.317 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T22:32:41.317 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T22:32:41.319 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T22:32:41.322 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T22:32:41.364 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:32:41.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:32:41.466 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:32:41.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:32:41.587 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:32:41.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T22:32:41.651 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:32:41.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:32:41.750 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:32:41.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:32:41.878 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:32:41.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T22:32:41.945 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:32:41.990 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:32:42.045 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:32:42.131 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:32:42.174 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:32:42.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T22:32:42.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-04-26T22:32:42.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-26T22:32:42.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-26T22:32:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:32:42.285 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:32:42.322 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:32:42.774 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T22:32:42.781 INFO:teuthology.task.internal:Starting timer... 2024-04-26T22:32:42.781 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T22:32:42.830 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T22:32:42.862 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T22:32:42.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-04-26T22:32:42.920 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-04-26T22:32:42.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-26T22:32:42.992 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-26T22:32:42.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-26T22:32:43.074 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-26T22:32:43.076 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T22:32:43.076 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-04-26T22:32:43.096 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-04-26T22:32:43.096 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-26T22:32:43.124 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-26T22:32:43.124 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-26T22:32:43.152 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-26T22:32:43.152 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-26T22:32:43.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:32:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:32:43.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:32:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:32:43.206 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:32:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:32:43.232 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T22:32:43.233 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-04-26T22:32:43.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-26T22:32:43.282 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-26T22:32:43.313 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T22:32:43.322 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T22:32:43.340 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T22:32:43.342 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi038.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T22:36:41.223 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-26T22:36:41.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T22:36:41.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:36:41.311 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-04-26T22:36:41.396 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T22:36:41.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-26T22:36:41.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:36:41.476 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-26T22:36:41.558 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-26T22:36:41.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T22:36:41.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:36:41.633 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-26T22:36:41.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T22:36:41.711 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T22:36:41.721 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T22:36:41.721 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T22:36:41.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:36:41.724 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T22:36:41.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:36:41.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T22:36:41.727 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:36:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T22:36:41.756 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T22:36:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T22:36:41.771 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T22:36:41.782 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T22:36:41.795 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-26T22:36:41.796 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T22:36:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-04-26T22:36:41.807 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-26T22:36:41.808 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-04-26T22:36:41.820 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-26T22:36:41.821 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T22:36:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T22:36:41.831 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-26T22:36:41.835 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T22:36:41.837 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T22:36:41.844 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T22:36:41.857 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T22:36:41.891 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:36:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:36:41.895 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:36:41.895 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-04-26T22:36:41.895 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.895 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.895 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.895 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.897 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:36:41.897 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-26T22:36:41.897 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.897 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.897 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.898 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.919 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:36:41.923 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:36:41.924 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-26T22:36:41.924 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.924 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.924 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.924 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:36:41.924 INFO:teuthology.run_tasks:Running task install... 2024-04-26T22:36:41.932 DEBUG:teuthology.task.install:project ceph 2024-04-26T22:36:41.932 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T22:36:41.932 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T22:36:41.932 INFO:teuthology.task.install:Using flavor: default 2024-04-26T22:36:41.938 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T22:36:41.938 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T22:36:41.939 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T22:36:41.939 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:36:41.941 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T22:36:41.941 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:36:41.943 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T22:36:41.943 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:36:42.086 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T22:36:42.086 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T22:36:42.090 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T22:36:42.091 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T22:36:42.158 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T22:36:42.158 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T22:36:42.207 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T22:36:42.207 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:36:42.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T22:36:42.233 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T22:36:42.233 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T22:36:42.262 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T22:36:42.262 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:36:42.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T22:36:42.288 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T22:36:42.289 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T22:36:42.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T22:36:42.329 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T22:36:42.330 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:36:42.330 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T22:36:42.355 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T22:36:42.355 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T22:36:42.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T22:36:42.372 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T22:36:42.421 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T22:36:42.431 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T22:36:42.435 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-26T22:36:42.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-26T22:36:42.494 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T22:36:42.495 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-04-26T22:36:42.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-04-26T22:36:42.558 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-26T22:36:42.559 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-26T22:36:42.671 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-26T22:36:42.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T22:36:42.731 INFO:teuthology.orchestra.run.smithi001.stdout:48 files removed 2024-04-26T22:36:42.758 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T22:36:42.783 INFO:teuthology.orchestra.run.smithi186.stdout:48 files removed 2024-04-26T22:36:42.810 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T22:36:43.389 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 200 kB/s | 84 kB 00:00 2024-04-26T22:36:43.482 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 203 kB/s | 84 kB 00:00 2024-04-26T22:36:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 163 kB/s | 84 kB 00:00 2024-04-26T22:36:43.774 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-04-26T22:36:43.859 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T22:36:43.922 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T22:36:44.209 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-26T22:36:44.258 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-04-26T22:36:44.289 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-04-26T22:36:44.775 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-26T22:36:44.784 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-26T22:36:46.651 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-26T22:36:48.105 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-04-26T22:36:50.351 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-26T22:36:54.791 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.2 MB 00:00 2024-04-26T22:36:56.525 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-26T22:36:57.510 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 885 kB/s | 6.2 MB 00:07 2024-04-26T22:36:57.560 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-26T22:36:59.219 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-26T22:37:00.507 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-26T22:37:02.827 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-04-26T22:37:05.028 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:37:05.029 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:37:05.040 INFO:teuthology.orchestra.run.smithi001.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T22:37:05.041 INFO:teuthology.orchestra.run.smithi001.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T22:37:05.041 INFO:teuthology.orchestra.run.smithi001.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T22:37:05.042 INFO:teuthology.orchestra.run.smithi001.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T22:37:05.044 INFO:teuthology.orchestra.run.smithi001.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T22:37:05.046 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T22:37:05.047 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T22:37:05.111 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-04-26T22:37:05.120 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-04-26T22:37:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-04-26T22:37:05.120 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-04-26T22:37:05.120 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T22:37:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T22:37:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-04-26T22:37:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T22:37:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T22:37:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T22:37:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T22:37:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T22:37:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T22:37:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T22:37:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T22:37:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T22:37:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T22:37:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:37:05.133 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-04-26T22:37:05.134 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-04-26T22:37:05.134 INFO:teuthology.orchestra.run.smithi001.stdout:Install 148 Packages 2024-04-26T22:37:05.134 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 4 Packages 2024-04-26T22:37:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:37:05.134 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 222 M 2024-04-26T22:37:05.134 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-04-26T22:37:05.784 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 977 kB/s | 19 MB 00:19 2024-04-26T22:37:05.923 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-26T22:37:06.228 INFO:teuthology.orchestra.run.smithi001.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-26T22:37:06.945 INFO:teuthology.orchestra.run.smithi001.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.2 MB/s | 889 kB 00:00 2024-04-26T22:37:07.020 INFO:teuthology.orchestra.run.smithi001.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.9 MB/s | 145 kB 00:00 2024-04-26T22:37:07.287 INFO:teuthology.orchestra.run.smithi001.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 3.8 MB/s | 5.4 MB 00:01 2024-04-26T22:37:07.546 INFO:teuthology.orchestra.run.smithi001.stdout:(5/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 4.5 MB/s | 2.4 MB 00:00 2024-04-26T22:37:07.588 INFO:teuthology.orchestra.run.smithi001.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-04-26T22:37:08.142 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:37:08.143 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:37:08.155 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T22:37:08.156 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T22:37:08.157 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T22:37:08.157 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T22:37:08.159 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T22:37:08.161 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T22:37:08.162 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T22:37:08.228 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-26T22:37:08.238 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T22:37:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T22:37:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:37:08.241 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T22:37:08.242 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T22:37:08.243 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T22:37:08.244 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T22:37:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T22:37:08.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T22:37:08.247 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T22:37:08.248 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T22:37:08.249 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T22:37:08.250 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout:Install 148 Packages 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 4 Packages 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 222 M 2024-04-26T22:37:08.251 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-26T22:37:08.330 INFO:teuthology.orchestra.run.smithi001.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 6.0 MB/s | 4.7 MB 00:00 2024-04-26T22:37:09.558 INFO:teuthology.orchestra.run.smithi038.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-26T22:37:09.621 INFO:teuthology.orchestra.run.smithi001.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 8.2 MB/s | 11 MB 00:01 2024-04-26T22:37:09.705 INFO:teuthology.orchestra.run.smithi001.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 302 kB/s | 25 kB 00:00 2024-04-26T22:37:10.375 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.2 MB 00:00 2024-04-26T22:37:10.734 INFO:teuthology.orchestra.run.smithi038.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 757 kB/s | 889 kB 00:01 2024-04-26T22:37:10.925 INFO:teuthology.orchestra.run.smithi038.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 758 kB/s | 145 kB 00:00 2024-04-26T22:37:11.155 INFO:teuthology.orchestra.run.smithi001.stdout:(10/152): ceph-common-19.0.0-3306.gbc612ffb.el9 4.0 MB/s | 21 MB 00:05 2024-04-26T22:37:11.230 INFO:teuthology.orchestra.run.smithi001.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.9 MB/s | 225 kB 00:00 2024-04-26T22:37:11.284 INFO:teuthology.orchestra.run.smithi038.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-04-26T22:37:11.306 INFO:teuthology.orchestra.run.smithi001.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 360 kB/s | 27 kB 00:00 2024-04-26T22:37:11.389 INFO:teuthology.orchestra.run.smithi001.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 381 kB/s | 32 kB 00:00 2024-04-26T22:37:11.598 INFO:teuthology.orchestra.run.smithi001.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 3.5 MB/s | 747 kB 00:00 2024-04-26T22:37:11.682 INFO:teuthology.orchestra.run.smithi001.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.9 MB/s | 163 kB 00:00 2024-04-26T22:37:11.692 INFO:teuthology.orchestra.run.smithi038.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.7 MB/s | 1.5 MB 00:00 2024-04-26T22:37:11.740 INFO:teuthology.orchestra.run.smithi001.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 2.1 MB/s | 127 kB 00:00 2024-04-26T22:37:11.851 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-26T22:37:11.882 INFO:teuthology.orchestra.run.smithi001.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 3.5 MB/s | 501 kB 00:00 2024-04-26T22:37:12.057 INFO:teuthology.orchestra.run.smithi001.stdout:(18/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 3.8 MB/s | 17 MB 00:04 2024-04-26T22:37:12.149 INFO:teuthology.orchestra.run.smithi001.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 494 kB/s | 45 kB 00:00 2024-04-26T22:37:12.233 INFO:teuthology.orchestra.run.smithi001.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.6 MB/s | 135 kB 00:00 2024-04-26T22:37:12.325 INFO:teuthology.orchestra.run.smithi001.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.7 MB/s | 161 kB 00:00 2024-04-26T22:37:12.492 INFO:teuthology.orchestra.run.smithi001.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 1.9 MB/s | 321 kB 00:00 2024-04-26T22:37:12.667 INFO:teuthology.orchestra.run.smithi001.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 298 kB 00:00 2024-04-26T22:37:12.759 INFO:teuthology.orchestra.run.smithi001.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.1 MB/s | 99 kB 00:00 2024-04-26T22:37:12.851 INFO:teuthology.orchestra.run.smithi001.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 932 kB/s | 85 kB 00:00 2024-04-26T22:37:12.910 INFO:teuthology.orchestra.run.smithi038.stdout:(6/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-04-26T22:37:13.043 INFO:teuthology.orchestra.run.smithi038.stdout:(7/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.1 MB/s | 2.4 MB 00:02 2024-04-26T22:37:13.210 INFO:teuthology.orchestra.run.smithi001.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 4.0 MB/s | 5.4 MB 00:01 2024-04-26T22:37:13.302 INFO:teuthology.orchestra.run.smithi001.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-26T22:37:13.385 INFO:teuthology.orchestra.run.smithi001.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 412 kB/s | 34 kB 00:00 2024-04-26T22:37:13.486 INFO:teuthology.orchestra.run.smithi001.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.4 MB/s | 142 kB 00:00 2024-04-26T22:37:13.769 INFO:teuthology.orchestra.run.smithi038.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 4.6 MB/s | 21 MB 00:04 2024-04-26T22:37:13.852 INFO:teuthology.orchestra.run.smithi038.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 303 kB/s | 25 kB 00:00 2024-04-26T22:37:14.061 INFO:teuthology.orchestra.run.smithi001.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 2.6 MB/s | 3.1 MB 00:01 2024-04-26T22:37:14.170 INFO:teuthology.orchestra.run.smithi001.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 5.2 MB/s | 3.5 MB 00:00 2024-04-26T22:37:14.270 INFO:teuthology.orchestra.run.smithi001.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.4 MB/s | 249 kB 00:00 2024-04-26T22:37:14.362 INFO:teuthology.orchestra.run.smithi001.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 538 kB/s | 49 kB 00:00 2024-04-26T22:37:14.454 INFO:teuthology.orchestra.run.smithi001.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 180 kB/s | 16 kB 00:00 2024-04-26T22:37:14.596 INFO:teuthology.orchestra.run.smithi001.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 1.9 MB/s | 273 kB 00:00 2024-04-26T22:37:14.838 INFO:teuthology.orchestra.run.smithi001.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.1 MB/s | 762 kB 00:00 2024-04-26T22:37:14.921 INFO:teuthology.orchestra.run.smithi001.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 219 kB/s | 18 kB 00:00 2024-04-26T22:37:14.961 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-04-26T22:37:15.172 INFO:teuthology.orchestra.run.smithi001.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 321 kB/s | 80 kB 00:00 2024-04-26T22:37:15.289 INFO:teuthology.orchestra.run.smithi001.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-04-26T22:37:15.339 INFO:teuthology.orchestra.run.smithi001.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-26T22:37:15.389 INFO:teuthology.orchestra.run.smithi001.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-26T22:37:15.473 INFO:teuthology.orchestra.run.smithi001.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-26T22:37:15.515 INFO:teuthology.orchestra.run.smithi001.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-26T22:37:15.556 INFO:teuthology.orchestra.run.smithi001.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-26T22:37:15.607 INFO:teuthology.orchestra.run.smithi001.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-26T22:37:15.657 INFO:teuthology.orchestra.run.smithi001.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-26T22:37:16.049 INFO:teuthology.orchestra.run.smithi001.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 689 kB 00:00 2024-04-26T22:37:16.400 INFO:teuthology.orchestra.run.smithi001.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-04-26T22:37:16.525 INFO:teuthology.orchestra.run.smithi001.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 857 kB/s | 107 kB 00:00 2024-04-26T22:37:16.575 INFO:teuthology.orchestra.run.smithi001.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-26T22:37:16.742 INFO:teuthology.orchestra.run.smithi001.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-04-26T22:37:16.784 INFO:teuthology.orchestra.run.smithi001.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 787 kB/s | 33 kB 00:00 2024-04-26T22:37:17.394 INFO:teuthology.orchestra.run.smithi038.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 3.8 MB/s | 17 MB 00:04 2024-04-26T22:37:17.485 INFO:teuthology.orchestra.run.smithi038.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.4 MB/s | 225 kB 00:00 2024-04-26T22:37:17.561 INFO:teuthology.orchestra.run.smithi038.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 361 kB/s | 27 kB 00:00 2024-04-26T22:37:17.636 INFO:teuthology.orchestra.run.smithi038.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 423 kB/s | 32 kB 00:00 2024-04-26T22:37:17.786 INFO:teuthology.orchestra.run.smithi038.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.9 MB/s | 747 kB 00:00 2024-04-26T22:37:17.918 INFO:teuthology.orchestra.run.smithi001.stdout:(53/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 6.0 MB/s | 49 MB 00:08 2024-04-26T22:37:18.010 INFO:teuthology.orchestra.run.smithi001.stdout:(54/152): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-04-26T22:37:18.060 INFO:teuthology.orchestra.run.smithi001.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-04-26T22:37:18.077 INFO:teuthology.orchestra.run.smithi001.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 115 kB/s | 18 kB 00:00 2024-04-26T22:37:18.102 INFO:teuthology.orchestra.run.smithi001.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-26T22:37:18.144 INFO:teuthology.orchestra.run.smithi001.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-04-26T22:37:18.169 INFO:teuthology.orchestra.run.smithi001.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 491 kB/s | 45 kB 00:00 2024-04-26T22:37:18.253 INFO:teuthology.orchestra.run.smithi038.stdout:(15/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 11 MB/s | 49 MB 00:04 2024-04-26T22:37:18.279 INFO:teuthology.orchestra.run.smithi038.stdout:(16/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 332 kB/s | 163 kB 00:00 2024-04-26T22:37:18.304 INFO:teuthology.orchestra.run.smithi038.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 2.5 MB/s | 127 kB 00:00 2024-04-26T22:37:18.311 INFO:teuthology.orchestra.run.smithi001.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 179 kB/s | 25 kB 00:00 2024-04-26T22:37:18.370 INFO:teuthology.orchestra.run.smithi001.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 2.5 MB/s | 582 kB 00:00 2024-04-26T22:37:18.371 INFO:teuthology.orchestra.run.smithi038.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 5.3 MB/s | 501 kB 00:00 2024-04-26T22:37:18.395 INFO:teuthology.orchestra.run.smithi001.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-04-26T22:37:18.420 INFO:teuthology.orchestra.run.smithi001.stdout:(63/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-04-26T22:37:18.487 INFO:teuthology.orchestra.run.smithi001.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-26T22:37:18.505 INFO:teuthology.orchestra.run.smithi038.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 338 kB/s | 45 kB 00:00 2024-04-26T22:37:18.521 INFO:teuthology.orchestra.run.smithi001.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 35 kB 00:00 2024-04-26T22:37:18.546 INFO:teuthology.orchestra.run.smithi001.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-04-26T22:37:18.588 INFO:teuthology.orchestra.run.smithi038.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.6 MB/s | 135 kB 00:00 2024-04-26T22:37:18.655 INFO:teuthology.orchestra.run.smithi038.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.4 MB/s | 161 kB 00:00 2024-04-26T22:37:18.739 INFO:teuthology.orchestra.run.smithi038.stdout:(22/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 12 MB/s | 5.4 MB 00:00 2024-04-26T22:37:18.773 INFO:teuthology.orchestra.run.smithi038.stdout:(23/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.7 MB/s | 321 kB 00:00 2024-04-26T22:37:18.806 INFO:teuthology.orchestra.run.smithi038.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 4.3 MB/s | 298 kB 00:00 2024-04-26T22:37:18.832 INFO:teuthology.orchestra.run.smithi038.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 99 kB 00:00 2024-04-26T22:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-26T22:37:18.897 INFO:teuthology.orchestra.run.smithi001.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-04-26T22:37:18.941 INFO:teuthology.orchestra.run.smithi038.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-26T22:37:19.017 INFO:teuthology.orchestra.run.smithi038.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 456 kB/s | 34 kB 00:00 2024-04-26T22:37:19.099 INFO:teuthology.orchestra.run.smithi038.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.7 MB/s | 142 kB 00:00 2024-04-26T22:37:19.147 INFO:teuthology.orchestra.run.smithi001.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 3.6 MB/s | 911 kB 00:00 2024-04-26T22:37:19.239 INFO:teuthology.orchestra.run.smithi001.stdout:(69/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.3 MB/s | 4.7 MB 00:00 2024-04-26T22:37:19.264 INFO:teuthology.orchestra.run.smithi001.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-04-26T22:37:19.275 INFO:teuthology.orchestra.run.smithi038.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 7.0 MB/s | 3.1 MB 00:00 2024-04-26T22:37:19.367 INFO:teuthology.orchestra.run.smithi038.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 13 MB/s | 3.5 MB 00:00 2024-04-26T22:37:19.473 INFO:teuthology.orchestra.run.smithi001.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-04-26T22:37:19.526 INFO:teuthology.orchestra.run.smithi038.stdout:(32/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 1.6 MB/s | 11 MB 00:06 2024-04-26T22:37:19.531 INFO:teuthology.orchestra.run.smithi001.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T22:37:19.551 INFO:teuthology.orchestra.run.smithi038.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3306.gbc 1.3 MB/s | 249 kB 00:00 2024-04-26T22:37:19.593 INFO:teuthology.orchestra.run.smithi038.stdout:(34/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 739 kB/s | 49 kB 00:00 2024-04-26T22:37:19.618 INFO:teuthology.orchestra.run.smithi038.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 246 kB/s | 16 kB 00:00 2024-04-26T22:37:19.718 INFO:teuthology.orchestra.run.smithi038.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 7.5 MB/s | 762 kB 00:00 2024-04-26T22:37:19.743 INFO:teuthology.orchestra.run.smithi038.stdout:(37/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 1.8 MB/s | 273 kB 00:00 2024-04-26T22:37:19.777 INFO:teuthology.orchestra.run.smithi038.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 311 kB/s | 18 kB 00:00 2024-04-26T22:37:19.806 INFO:teuthology.orchestra.run.smithi001.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-04-26T22:37:19.881 INFO:teuthology.orchestra.run.smithi001.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 9.0 MB/s | 5.8 MB 00:00 2024-04-26T22:37:19.940 INFO:teuthology.orchestra.run.smithi001.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-26T22:37:19.952 INFO:teuthology.orchestra.run.smithi038.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 385 kB/s | 80 kB 00:00 2024-04-26T22:37:19.965 INFO:teuthology.orchestra.run.smithi001.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2024-04-26T22:37:20.016 INFO:teuthology.orchestra.run.smithi001.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 341 kB/s | 26 kB 00:00 2024-04-26T22:37:20.019 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-26T22:37:20.028 INFO:teuthology.orchestra.run.smithi038.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-26T22:37:20.041 INFO:teuthology.orchestra.run.smithi001.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 255 kB/s | 19 kB 00:00 2024-04-26T22:37:20.061 INFO:teuthology.orchestra.run.smithi038.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-04-26T22:37:20.074 INFO:teuthology.orchestra.run.smithi001.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T22:37:20.100 INFO:teuthology.orchestra.run.smithi001.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 592 kB/s | 35 kB 00:00 2024-04-26T22:37:20.103 INFO:teuthology.orchestra.run.smithi038.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-26T22:37:20.162 INFO:teuthology.orchestra.run.smithi038.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-04-26T22:37:20.208 INFO:teuthology.orchestra.run.smithi001.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-26T22:37:20.254 INFO:teuthology.orchestra.run.smithi038.stdout:(44/152): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-04-26T22:37:20.279 INFO:teuthology.orchestra.run.smithi038.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 606 kB/s | 106 kB 00:00 2024-04-26T22:37:20.321 INFO:teuthology.orchestra.run.smithi038.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 852 kB/s | 135 kB 00:00 2024-04-26T22:37:20.380 INFO:teuthology.orchestra.run.smithi038.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-26T22:37:20.467 INFO:teuthology.orchestra.run.smithi001.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 860 kB/s | 222 kB 00:00 2024-04-26T22:37:20.480 INFO:teuthology.orchestra.run.smithi038.stdout:(48/152): python3-requests-2.25.1-8.el9.noarch. 624 kB/s | 125 kB 00:00 2024-04-26T22:37:20.517 INFO:teuthology.orchestra.run.smithi001.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T22:37:20.606 INFO:teuthology.orchestra.run.smithi038.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-04-26T22:37:20.610 INFO:teuthology.orchestra.run.smithi001.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.5 MB/s | 5.1 MB 00:00 2024-04-26T22:37:20.635 INFO:teuthology.orchestra.run.smithi001.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 361 kB/s | 42 kB 00:00 2024-04-26T22:37:20.664 INFO:teuthology.orchestra.run.smithi038.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 369 kB/s | 21 kB 00:00 2024-04-26T22:37:20.685 INFO:teuthology.orchestra.run.smithi001.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-26T22:37:20.690 INFO:teuthology.orchestra.run.smithi038.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 511 kB/s | 107 kB 00:00 2024-04-26T22:37:20.727 INFO:teuthology.orchestra.run.smithi001.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-26T22:37:20.731 INFO:teuthology.orchestra.run.smithi038.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-04-26T22:37:20.752 INFO:teuthology.orchestra.run.smithi001.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-26T22:37:20.756 INFO:teuthology.orchestra.run.smithi038.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 489 kB/s | 33 kB 00:00 2024-04-26T22:37:20.790 INFO:teuthology.orchestra.run.smithi038.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-04-26T22:37:20.807 INFO:teuthology.orchestra.run.smithi038.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 361 kB/s | 18 kB 00:00 2024-04-26T22:37:20.832 INFO:teuthology.orchestra.run.smithi038.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 265 kB/s | 11 kB 00:00 2024-04-26T22:37:20.836 INFO:teuthology.orchestra.run.smithi001.stdout:(89/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-04-26T22:37:20.853 INFO:teuthology.orchestra.run.smithi001.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 407 kB/s | 51 kB 00:00 2024-04-26T22:37:20.869 INFO:teuthology.orchestra.run.smithi001.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 457 kB/s | 54 kB 00:00 2024-04-26T22:37:20.883 INFO:teuthology.orchestra.run.smithi038.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-26T22:37:20.895 INFO:teuthology.orchestra.run.smithi001.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 993 kB/s | 42 kB 00:00 2024-04-26T22:37:20.899 INFO:teuthology.orchestra.run.smithi038.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 679 kB/s | 45 kB 00:00 2024-04-26T22:37:20.920 INFO:teuthology.orchestra.run.smithi001.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 881 kB/s | 44 kB 00:00 2024-04-26T22:37:20.953 INFO:teuthology.orchestra.run.smithi001.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-26T22:37:20.981 INFO:teuthology.orchestra.run.smithi001.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 592 kB/s | 36 kB 00:00 2024-04-26T22:37:21.002 INFO:teuthology.orchestra.run.smithi038.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-26T22:37:21.006 INFO:teuthology.orchestra.run.smithi001.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 907 kB/s | 47 kB 00:00 2024-04-26T22:37:21.057 INFO:teuthology.orchestra.run.smithi001.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T22:37:21.115 INFO:teuthology.orchestra.run.smithi001.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-26T22:37:21.141 INFO:teuthology.orchestra.run.smithi001.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-04-26T22:37:21.166 INFO:teuthology.orchestra.run.smithi001.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T22:37:21.191 INFO:teuthology.orchestra.run.smithi001.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-04-26T22:37:21.216 INFO:teuthology.orchestra.run.smithi001.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-04-26T22:37:21.225 INFO:teuthology.orchestra.run.smithi038.stdout:(60/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.1 MB/s | 3.0 MB 00:00 2024-04-26T22:37:21.251 INFO:teuthology.orchestra.run.smithi038.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 102 kB/s | 25 kB 00:00 2024-04-26T22:37:21.276 INFO:teuthology.orchestra.run.smithi038.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-04-26T22:37:21.301 INFO:teuthology.orchestra.run.smithi038.stdout:(63/152): libtool-2.4.6-45.el9.x86_64.rpm 1.4 MB/s | 582 kB 00:00 2024-04-26T22:37:21.326 INFO:teuthology.orchestra.run.smithi038.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-26T22:37:21.352 INFO:teuthology.orchestra.run.smithi038.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-26T22:37:21.377 INFO:teuthology.orchestra.run.smithi038.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-26T22:37:21.502 INFO:teuthology.orchestra.run.smithi038.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-26T22:37:21.541 INFO:teuthology.orchestra.run.smithi001.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-04-26T22:37:21.578 INFO:teuthology.orchestra.run.smithi038.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-26T22:37:21.628 INFO:teuthology.orchestra.run.smithi038.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-04-26T22:37:21.753 INFO:teuthology.orchestra.run.smithi038.stdout:(70/152): postgresql-13.14-1.el9.x86_64.rpm 4.2 MB/s | 1.6 MB 00:00 2024-04-26T22:37:21.809 INFO:teuthology.orchestra.run.smithi001.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 926 kB/s | 548 kB 00:00 2024-04-26T22:37:21.812 INFO:teuthology.orchestra.run.smithi038.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-26T22:37:21.834 INFO:teuthology.orchestra.run.smithi001.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-04-26T22:37:21.876 INFO:teuthology.orchestra.run.smithi001.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-04-26T22:37:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-26T22:37:21.913 INFO:teuthology.orchestra.run.smithi038.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 873 kB/s | 87 kB 00:00 2024-04-26T22:37:21.979 INFO:teuthology.orchestra.run.smithi038.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-26T22:37:22.013 INFO:teuthology.orchestra.run.smithi038.stdout:(75/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-04-26T22:37:22.038 INFO:teuthology.orchestra.run.smithi038.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-04-26T22:37:22.064 INFO:teuthology.orchestra.run.smithi038.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 508 kB/s | 26 kB 00:00 2024-04-26T22:37:22.068 INFO:teuthology.orchestra.run.smithi001.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 136 kB/s | 26 kB 00:00 2024-04-26T22:37:22.089 INFO:teuthology.orchestra.run.smithi038.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-04-26T22:37:22.132 INFO:teuthology.orchestra.run.smithi038.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-26T22:37:22.143 INFO:teuthology.orchestra.run.smithi001.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T22:37:22.145 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:37:22.146 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:37:22.156 INFO:teuthology.orchestra.run.smithi038.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-04-26T22:37:22.156 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T22:37:22.157 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T22:37:22.158 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T22:37:22.159 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T22:37:22.160 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T22:37:22.162 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T22:37:22.163 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T22:37:22.168 INFO:teuthology.orchestra.run.smithi001.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 489 kB/s | 49 kB 00:00 2024-04-26T22:37:22.210 INFO:teuthology.orchestra.run.smithi001.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-26T22:37:22.223 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-26T22:37:22.240 INFO:teuthology.orchestra.run.smithi038.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T22:37:22.244 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T22:37:22.245 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T22:37:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T22:37:22.247 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T22:37:22.248 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:37:22.249 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T22:37:22.250 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T22:37:22.251 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T22:37:22.252 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T22:37:22.253 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T22:37:22.254 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T22:37:22.255 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 4 Packages 2024-04-26T22:37:22.256 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:37:22.257 INFO:teuthology.orchestra.run.smithi001.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-26T22:37:22.257 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 222 M 2024-04-26T22:37:22.257 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-26T22:37:22.265 INFO:teuthology.orchestra.run.smithi038.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.2 MB/s | 250 kB 00:00 2024-04-26T22:37:22.290 INFO:teuthology.orchestra.run.smithi038.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-26T22:37:22.303 INFO:teuthology.orchestra.run.smithi001.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-04-26T22:37:22.316 INFO:teuthology.orchestra.run.smithi038.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T22:37:22.336 INFO:teuthology.orchestra.run.smithi001.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-04-26T22:37:22.341 INFO:teuthology.orchestra.run.smithi038.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-26T22:37:22.366 INFO:teuthology.orchestra.run.smithi038.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-26T22:37:22.378 INFO:teuthology.orchestra.run.smithi001.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 587 kB/s | 43 kB 00:00 2024-04-26T22:37:22.391 INFO:teuthology.orchestra.run.smithi038.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-26T22:37:22.403 INFO:teuthology.orchestra.run.smithi001.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 480 kB/s | 32 kB 00:00 2024-04-26T22:37:22.417 INFO:teuthology.orchestra.run.smithi038.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-26T22:37:22.442 INFO:teuthology.orchestra.run.smithi038.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T22:37:22.445 INFO:teuthology.orchestra.run.smithi001.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-26T22:37:22.459 INFO:teuthology.orchestra.run.smithi038.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-04-26T22:37:22.479 INFO:teuthology.orchestra.run.smithi001.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T22:37:22.510 INFO:teuthology.orchestra.run.smithi038.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-04-26T22:37:22.546 INFO:teuthology.orchestra.run.smithi001.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-26T22:37:22.568 INFO:teuthology.orchestra.run.smithi038.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 759 kB/s | 44 kB 00:00 2024-04-26T22:37:22.571 INFO:teuthology.orchestra.run.smithi001.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-26T22:37:22.622 INFO:teuthology.orchestra.run.smithi001.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 472 kB/s | 35 kB 00:00 2024-04-26T22:37:22.643 INFO:teuthology.orchestra.run.smithi038.stdout:(93/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-26T22:37:22.646 INFO:teuthology.orchestra.run.smithi001.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-26T22:37:22.669 INFO:teuthology.orchestra.run.smithi038.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.2 MB/s | 221 kB 00:00 2024-04-26T22:37:22.686 INFO:teuthology.orchestra.run.smithi038.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 854 kB/s | 36 kB 00:00 2024-04-26T22:37:22.719 INFO:teuthology.orchestra.run.smithi038.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-26T22:37:22.744 INFO:teuthology.orchestra.run.smithi038.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-26T22:37:22.747 INFO:teuthology.orchestra.run.smithi001.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-04-26T22:37:22.770 INFO:teuthology.orchestra.run.smithi038.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T22:37:22.772 INFO:teuthology.orchestra.run.smithi001.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-26T22:37:22.796 INFO:teuthology.orchestra.run.smithi038.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-26T22:37:22.806 INFO:teuthology.orchestra.run.smithi001.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-26T22:37:22.821 INFO:teuthology.orchestra.run.smithi038.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T22:37:22.831 INFO:teuthology.orchestra.run.smithi001.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 300 kB/s | 18 kB 00:00 2024-04-26T22:37:22.846 INFO:teuthology.orchestra.run.smithi038.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-26T22:37:22.865 INFO:teuthology.orchestra.run.smithi001.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-04-26T22:37:22.907 INFO:teuthology.orchestra.run.smithi001.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-26T22:37:22.932 INFO:teuthology.orchestra.run.smithi001.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 293 kB/s | 20 kB 00:00 2024-04-26T22:37:22.938 INFO:teuthology.orchestra.run.smithi038.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-04-26T22:37:22.999 INFO:teuthology.orchestra.run.smithi001.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-26T22:37:23.024 INFO:teuthology.orchestra.run.smithi001.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-04-26T22:37:23.066 INFO:teuthology.orchestra.run.smithi001.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-26T22:37:23.091 INFO:teuthology.orchestra.run.smithi001.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 862 kB/s | 58 kB 00:00 2024-04-26T22:37:23.097 INFO:teuthology.orchestra.run.smithi038.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-26T22:37:23.142 INFO:teuthology.orchestra.run.smithi001.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-26T22:37:23.167 INFO:teuthology.orchestra.run.smithi001.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-26T22:37:23.197 INFO:teuthology.orchestra.run.smithi038.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-04-26T22:37:23.212 INFO:teuthology.orchestra.run.smithi186.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-26T22:37:23.317 INFO:teuthology.orchestra.run.smithi001.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 513 kB/s | 90 kB 00:00 2024-04-26T22:37:23.343 INFO:teuthology.orchestra.run.smithi001.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 161 kB/s | 28 kB 00:00 2024-04-26T22:37:23.385 INFO:teuthology.orchestra.run.smithi001.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-04-26T22:37:23.418 INFO:teuthology.orchestra.run.smithi001.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-26T22:37:23.452 INFO:teuthology.orchestra.run.smithi001.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-04-26T22:37:23.477 INFO:teuthology.orchestra.run.smithi001.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-04-26T22:37:23.496 INFO:teuthology.orchestra.run.smithi186.stdout:(2/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 10 MB/s | 5.4 MB 00:00 2024-04-26T22:37:23.511 INFO:teuthology.orchestra.run.smithi001.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 870 kB/s | 51 kB 00:00 2024-04-26T22:37:23.546 INFO:teuthology.orchestra.run.smithi186.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.8 MB/s | 145 kB 00:00 2024-04-26T22:37:23.553 INFO:teuthology.orchestra.run.smithi001.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T22:37:23.578 INFO:teuthology.orchestra.run.smithi001.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-26T22:37:23.628 INFO:teuthology.orchestra.run.smithi001.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-26T22:37:23.647 INFO:teuthology.orchestra.run.smithi038.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 43 kB/s | 19 kB 00:00 2024-04-26T22:37:23.653 INFO:teuthology.orchestra.run.smithi001.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-26T22:37:23.704 INFO:teuthology.orchestra.run.smithi001.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-26T22:37:23.704 INFO:teuthology.orchestra.run.smithi186.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 2.4 MB 00:00 2024-04-26T22:37:23.754 INFO:teuthology.orchestra.run.smithi001.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-26T22:37:23.813 INFO:teuthology.orchestra.run.smithi186.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 14 MB/s | 1.5 MB 00:00 2024-04-26T22:37:23.838 INFO:teuthology.orchestra.run.smithi186.stdout:(6/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 889 kB 00:00 2024-04-26T22:37:23.973 INFO:teuthology.orchestra.run.smithi038.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 364 kB/s | 309 kB 00:00 2024-04-26T22:37:24.004 INFO:teuthology.orchestra.run.smithi001.stdout:(148/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.1 MB/s | 16 MB 00:03 2024-04-26T22:37:24.081 INFO:teuthology.orchestra.run.smithi038.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-26T22:37:24.130 INFO:teuthology.orchestra.run.smithi186.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 4.7 MB 00:00 2024-04-26T22:37:24.148 INFO:teuthology.orchestra.run.smithi038.stdout:(108/152): abseil-cpp-20211102.0-3.el9.x86_64.r 453 kB/s | 548 kB 00:01 2024-04-26T22:37:24.182 INFO:teuthology.orchestra.run.smithi038.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-04-26T22:37:24.338 INFO:teuthology.orchestra.run.smithi001.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 1.3 MB 00:00 2024-04-26T22:37:24.340 INFO:teuthology.orchestra.run.smithi038.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-04-26T22:37:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 584 kB/s | 151 kB 00:00 2024-04-26T22:37:24.513 INFO:teuthology.orchestra.run.smithi001.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 4.2 MB/s | 3.4 MB 00:00 2024-04-26T22:37:24.805 INFO:teuthology.orchestra.run.smithi186.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 16 MB/s | 11 MB 00:00 2024-04-26T22:37:24.855 INFO:teuthology.orchestra.run.smithi186.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 506 kB/s | 25 kB 00:00 2024-04-26T22:37:24.997 INFO:teuthology.orchestra.run.smithi001.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.6 MB/s | 2.4 MB 00:00 2024-04-26T22:37:25.047 INFO:teuthology.orchestra.run.smithi001.stdout:(152/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-04-26T22:37:25.051 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:37:25.051 INFO:teuthology.orchestra.run.smithi001.stdout:Total 11 MB/s | 222 MB 00:19 2024-04-26T22:37:25.199 INFO:teuthology.orchestra.run.smithi038.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 723 kB/s | 548 kB 00:00 2024-04-26T22:37:25.308 INFO:teuthology.orchestra.run.smithi038.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-26T22:37:25.416 INFO:teuthology.orchestra.run.smithi038.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-26T22:37:25.450 INFO:teuthology.orchestra.run.smithi038.stdout:(115/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 756 kB/s | 839 kB 00:01 2024-04-26T22:37:25.517 INFO:teuthology.orchestra.run.smithi038.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-26T22:37:25.717 INFO:teuthology.orchestra.run.smithi038.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 644 kB/s | 172 kB 00:00 2024-04-26T22:37:26.059 INFO:teuthology.orchestra.run.smithi038.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 661 kB/s | 358 kB 00:00 2024-04-26T22:37:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-04-26T22:37:26.171 INFO:teuthology.orchestra.run.smithi038.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-04-26T22:37:26.193 INFO:teuthology.orchestra.run.smithi038.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 637 kB/s | 303 kB 00:00 2024-04-26T22:37:26.328 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-04-26T22:37:26.328 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-04-26T22:37:26.535 INFO:teuthology.orchestra.run.smithi038.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 560 kB/s | 206 kB 00:00 2024-04-26T22:37:26.819 INFO:teuthology.orchestra.run.smithi038.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 515 kB/s | 144 kB 00:00 2024-04-26T22:37:26.919 INFO:teuthology.orchestra.run.smithi038.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-26T22:37:27.020 INFO:teuthology.orchestra.run.smithi038.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-26T22:37:27.120 INFO:teuthology.orchestra.run.smithi038.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2024-04-26T22:37:27.220 INFO:teuthology.orchestra.run.smithi038.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-26T22:37:27.346 INFO:teuthology.orchestra.run.smithi038.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2024-04-26T22:37:27.922 INFO:teuthology.orchestra.run.smithi186.stdout:(10/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 16 MB/s | 49 MB 00:03 2024-04-26T22:37:28.081 INFO:teuthology.orchestra.run.smithi186.stdout:(11/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 4.0 MB/s | 17 MB 00:04 2024-04-26T22:37:28.106 INFO:teuthology.orchestra.run.smithi186.stdout:(12/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 1.2 MB/s | 225 kB 00:00 2024-04-26T22:37:28.156 INFO:teuthology.orchestra.run.smithi186.stdout:(13/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 361 kB/s | 27 kB 00:00 2024-04-26T22:37:28.198 INFO:teuthology.orchestra.run.smithi186.stdout:(14/152): libcephfs-devel-19.0.0-3306.gbc612ffb 345 kB/s | 32 kB 00:00 2024-04-26T22:37:28.210 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-04-26T22:37:28.210 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-04-26T22:37:28.257 INFO:teuthology.orchestra.run.smithi186.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.7 MB/s | 163 kB 00:00 2024-04-26T22:37:28.316 INFO:teuthology.orchestra.run.smithi186.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 2.1 MB/s | 127 kB 00:00 2024-04-26T22:37:28.349 INFO:teuthology.orchestra.run.smithi186.stdout:(17/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 3.8 MB/s | 747 kB 00:00 2024-04-26T22:37:28.558 INFO:teuthology.orchestra.run.smithi186.stdout:(18/152): ceph-common-19.0.0-3306.gbc612ffb.el9 3.8 MB/s | 21 MB 00:05 2024-04-26T22:37:28.592 INFO:teuthology.orchestra.run.smithi186.stdout:(19/152): libradosstriper1-19.0.0-3306.gbc612ff 1.8 MB/s | 501 kB 00:00 2024-04-26T22:37:28.642 INFO:teuthology.orchestra.run.smithi186.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 2.7 MB/s | 135 kB 00:00 2024-04-26T22:37:28.667 INFO:teuthology.orchestra.run.smithi186.stdout:(21/152): python3-ceph-argparse-19.0.0-3306.gbc 416 kB/s | 45 kB 00:00 2024-04-26T22:37:28.701 INFO:teuthology.orchestra.run.smithi186.stdout:(22/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.7 MB/s | 161 kB 00:00 2024-04-26T22:37:28.721 INFO:teuthology.orchestra.run.smithi038.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 760 kB/s | 1.0 MB 00:01 2024-04-26T22:37:28.759 INFO:teuthology.orchestra.run.smithi186.stdout:(23/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.4 MB/s | 321 kB 00:00 2024-04-26T22:37:28.784 INFO:teuthology.orchestra.run.smithi186.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.5 MB/s | 298 kB 00:00 2024-04-26T22:37:28.826 INFO:teuthology.orchestra.run.smithi186.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T22:37:28.838 INFO:teuthology.orchestra.run.smithi038.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 789 kB/s | 2.0 MB 00:02 2024-04-26T22:37:28.851 INFO:teuthology.orchestra.run.smithi186.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-26T22:37:28.863 INFO:teuthology.orchestra.run.smithi038.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-04-26T22:37:28.912 INFO:teuthology.orchestra.run.smithi186.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.8 MB/s | 171 kB 00:00 2024-04-26T22:37:28.988 INFO:teuthology.orchestra.run.smithi186.stdout:(28/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 8.4 MB/s | 5.4 MB 00:00 2024-04-26T22:37:29.004 INFO:teuthology.orchestra.run.smithi186.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3306.g 373 kB/s | 34 kB 00:00 2024-04-26T22:37:29.021 INFO:teuthology.orchestra.run.smithi038.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 430 kB/s | 79 kB 00:00 2024-04-26T22:37:29.039 INFO:teuthology.orchestra.run.smithi038.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 329 kB/s | 58 kB 00:00 2024-04-26T22:37:29.055 INFO:teuthology.orchestra.run.smithi186.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.1 MB/s | 142 kB 00:00 2024-04-26T22:37:29.148 INFO:teuthology.orchestra.run.smithi038.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-26T22:37:29.255 INFO:teuthology.orchestra.run.smithi186.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 14 MB/s | 3.5 MB 00:00 2024-04-26T22:37:29.273 INFO:teuthology.orchestra.run.smithi038.stdout:(134/152): libarrow-9.0.0-11.el9.x86_64.rpm 804 kB/s | 4.4 MB 00:05 2024-04-26T22:37:29.314 INFO:teuthology.orchestra.run.smithi186.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 4.2 MB/s | 249 kB 00:00 2024-04-26T22:37:29.372 INFO:teuthology.orchestra.run.smithi186.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 851 kB/s | 49 kB 00:00 2024-04-26T22:37:29.381 INFO:teuthology.orchestra.run.smithi038.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 385 kB/s | 90 kB 00:00 2024-04-26T22:37:29.406 INFO:teuthology.orchestra.run.smithi038.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 211 kB/s | 28 kB 00:00 2024-04-26T22:37:29.422 INFO:teuthology.orchestra.run.smithi186.stdout:(34/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 5.2 MB/s | 3.1 MB 00:00 2024-04-26T22:37:29.448 INFO:teuthology.orchestra.run.smithi186.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 219 kB/s | 16 kB 00:00 2024-04-26T22:37:29.490 INFO:teuthology.orchestra.run.smithi038.stdout:(137/152): python3-pecan-1.4.2-3.el9.noarch.rpm 581 kB/s | 272 kB 00:00 2024-04-26T22:37:29.516 INFO:teuthology.orchestra.run.smithi038.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 230 kB/s | 31 kB 00:00 2024-04-26T22:37:29.523 INFO:teuthology.orchestra.run.smithi186.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 10 MB/s | 762 kB 00:00 2024-04-26T22:37:29.548 INFO:teuthology.orchestra.run.smithi186.stdout:(37/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.1 MB/s | 273 kB 00:00 2024-04-26T22:37:29.573 INFO:teuthology.orchestra.run.smithi186.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 364 kB/s | 18 kB 00:00 2024-04-26T22:37:29.632 INFO:teuthology.orchestra.run.smithi038.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 307 kB/s | 36 kB 00:00 2024-04-26T22:37:29.699 INFO:teuthology.orchestra.run.smithi038.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-04-26T22:37:29.733 INFO:teuthology.orchestra.run.smithi038.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 578 kB/s | 188 kB 00:00 2024-04-26T22:37:29.740 INFO:teuthology.orchestra.run.smithi186.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 419 kB/s | 80 kB 00:00 2024-04-26T22:37:29.758 INFO:teuthology.orchestra.run.smithi038.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 406 kB/s | 51 kB 00:00 2024-04-26T22:37:29.824 INFO:teuthology.orchestra.run.smithi186.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 9.6 MB/s | 7.4 MB 00:00 2024-04-26T22:37:29.891 INFO:teuthology.orchestra.run.smithi186.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-26T22:37:29.984 INFO:teuthology.orchestra.run.smithi038.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 358 kB/s | 90 kB 00:00 2024-04-26T22:37:30.091 INFO:teuthology.orchestra.run.smithi186.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-26T22:37:30.134 INFO:teuthology.orchestra.run.smithi038.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-04-26T22:37:30.158 INFO:teuthology.orchestra.run.smithi186.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 759 kB/s | 253 kB 00:00 2024-04-26T22:37:30.168 INFO:teuthology.orchestra.run.smithi038.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 491 kB/s | 230 kB 00:00 2024-04-26T22:37:30.242 INFO:teuthology.orchestra.run.smithi186.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-26T22:37:30.317 INFO:teuthology.orchestra.run.smithi186.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-26T22:37:30.384 INFO:teuthology.orchestra.run.smithi186.stdout:(46/152): python3-ply-3.11-14.el9.noarch.rpm 364 kB/s | 106 kB 00:00 2024-04-26T22:37:30.443 INFO:teuthology.orchestra.run.smithi186.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-26T22:37:30.476 INFO:teuthology.orchestra.run.smithi186.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 2.1 MB/s | 1.2 MB 00:00 2024-04-26T22:37:30.551 INFO:teuthology.orchestra.run.smithi038.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 539 kB/s | 427 kB 00:00 2024-04-26T22:37:30.577 INFO:teuthology.orchestra.run.smithi038.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 433 kB/s | 192 kB 00:00 2024-04-26T22:37:30.851 INFO:teuthology.orchestra.run.smithi186.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 285 kB/s | 107 kB 00:00 2024-04-26T22:37:30.885 INFO:teuthology.orchestra.run.smithi186.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 689 kB 00:00 2024-04-26T22:37:30.927 INFO:teuthology.orchestra.run.smithi186.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 1.9 MB/s | 943 kB 00:00 2024-04-26T22:37:30.952 INFO:teuthology.orchestra.run.smithi186.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-04-26T22:37:30.979 INFO:teuthology.orchestra.run.smithi186.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-26T22:37:31.002 INFO:teuthology.orchestra.run.smithi186.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-04-26T22:37:31.054 INFO:teuthology.orchestra.run.smithi186.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-04-26T22:37:31.128 INFO:teuthology.orchestra.run.smithi186.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 89 kB/s | 11 kB 00:00 2024-04-26T22:37:31.178 INFO:teuthology.orchestra.run.smithi186.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-26T22:37:31.204 INFO:teuthology.orchestra.run.smithi186.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-04-26T22:37:31.227 INFO:teuthology.orchestra.run.smithi038.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 5.0 MB/s | 3.4 MB 00:00 2024-04-26T22:37:31.254 INFO:teuthology.orchestra.run.smithi186.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 360 kB/s | 45 kB 00:00 2024-04-26T22:37:31.379 INFO:teuthology.orchestra.run.smithi186.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-04-26T22:37:31.405 INFO:teuthology.orchestra.run.smithi186.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 169 kB/s | 25 kB 00:00 2024-04-26T22:37:31.430 INFO:teuthology.orchestra.run.smithi186.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 2.5 MB/s | 582 kB 00:00 2024-04-26T22:37:31.455 INFO:teuthology.orchestra.run.smithi186.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-04-26T22:37:31.480 INFO:teuthology.orchestra.run.smithi186.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-26T22:37:31.506 INFO:teuthology.orchestra.run.smithi186.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-26T22:37:31.531 INFO:teuthology.orchestra.run.smithi186.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-26T22:37:31.569 INFO:teuthology.orchestra.run.smithi038.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.0 MB/s | 1.3 MB 00:00 2024-04-26T22:37:31.723 INFO:teuthology.orchestra.run.smithi186.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-26T22:37:31.769 INFO:teuthology.orchestra.run.smithi038.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-26T22:37:31.790 INFO:teuthology.orchestra.run.smithi186.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-26T22:37:31.857 INFO:teuthology.orchestra.run.smithi186.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-26T22:37:31.882 INFO:teuthology.orchestra.run.smithi186.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-26T22:37:31.974 INFO:teuthology.orchestra.run.smithi186.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-26T22:37:32.016 INFO:teuthology.orchestra.run.smithi186.stdout:(72/152): postgresql-13.14-1.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-04-26T22:37:32.019 INFO:teuthology.orchestra.run.smithi038.stdout:(151/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-04-26T22:37:32.074 INFO:teuthology.orchestra.run.smithi186.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 871 kB/s | 87 kB 00:00 2024-04-26T22:37:32.149 INFO:teuthology.orchestra.run.smithi186.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-04-26T22:37:32.200 INFO:teuthology.orchestra.run.smithi186.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-04-26T22:37:32.283 INFO:teuthology.orchestra.run.smithi186.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-04-26T22:37:32.359 INFO:teuthology.orchestra.run.smithi186.stdout:(77/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-26T22:37:32.384 INFO:teuthology.orchestra.run.smithi186.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 139 kB/s | 26 kB 00:00 2024-04-26T22:37:32.409 INFO:teuthology.orchestra.run.smithi186.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 152 kB/s | 19 kB 00:00 2024-04-26T22:37:32.426 INFO:teuthology.orchestra.run.smithi186.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-26T22:37:32.460 INFO:teuthology.orchestra.run.smithi186.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 461 kB/s | 35 kB 00:00 2024-04-26T22:37:32.518 INFO:teuthology.orchestra.run.smithi186.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-26T22:37:32.577 INFO:teuthology.orchestra.run.smithi038.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 672 kB/s | 1.6 MB 00:02 2024-04-26T22:37:32.586 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:37:32.586 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.1 MB/s | 222 MB 00:24 2024-04-26T22:37:32.610 INFO:teuthology.orchestra.run.smithi186.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-26T22:37:32.636 INFO:teuthology.orchestra.run.smithi186.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-04-26T22:37:32.660 INFO:teuthology.orchestra.run.smithi186.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 544 kB/s | 77 kB 00:00 2024-04-26T22:37:32.706 INFO:teuthology.orchestra.run.smithi186.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 460 kB/s | 42 kB 00:00 2024-04-26T22:37:32.728 INFO:teuthology.orchestra.run.smithi186.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-26T22:37:32.753 INFO:teuthology.orchestra.run.smithi186.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-26T22:37:32.787 INFO:teuthology.orchestra.run.smithi186.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.2 MB/s | 279 kB 00:00 2024-04-26T22:37:32.812 INFO:teuthology.orchestra.run.smithi186.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-04-26T22:37:32.837 INFO:teuthology.orchestra.run.smithi186.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 640 kB/s | 54 kB 00:00 2024-04-26T22:37:32.887 INFO:teuthology.orchestra.run.smithi186.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-04-26T22:37:32.955 INFO:teuthology.orchestra.run.smithi186.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 378 kB/s | 44 kB 00:00 2024-04-26T22:37:33.021 INFO:teuthology.orchestra.run.smithi186.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-04-26T22:37:33.047 INFO:teuthology.orchestra.run.smithi186.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 391 kB/s | 36 kB 00:00 2024-04-26T22:37:33.114 INFO:teuthology.orchestra.run.smithi186.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 515 kB/s | 47 kB 00:00 2024-04-26T22:37:33.281 INFO:teuthology.orchestra.run.smithi186.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-04-26T22:37:33.306 INFO:teuthology.orchestra.run.smithi186.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-04-26T22:37:33.331 INFO:teuthology.orchestra.run.smithi186.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 266 kB/s | 58 kB 00:00 2024-04-26T22:37:33.356 INFO:teuthology.orchestra.run.smithi186.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-26T22:37:33.417 INFO:teuthology.orchestra.run.smithi186.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 577 kB/s | 64 kB 00:00 2024-04-26T22:37:33.440 INFO:teuthology.orchestra.run.smithi186.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 412 kB/s | 45 kB 00:00 2024-04-26T22:37:33.465 INFO:teuthology.orchestra.run.smithi186.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-04-26T22:37:33.507 INFO:teuthology.orchestra.run.smithi186.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.3 MB/s | 863 kB 00:00 2024-04-26T22:37:33.708 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-26T22:37:33.868 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-26T22:37:33.868 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-26T22:37:33.983 INFO:teuthology.orchestra.run.smithi186.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 41 kB/s | 19 kB 00:00 2024-04-26T22:37:34.367 INFO:teuthology.orchestra.run.smithi186.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 343 kB/s | 309 kB 00:00 2024-04-26T22:37:34.399 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-04-26T22:37:34.475 INFO:teuthology.orchestra.run.smithi186.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-26T22:37:34.534 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T22:37:34.608 INFO:teuthology.orchestra.run.smithi186.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-04-26T22:37:34.717 INFO:teuthology.orchestra.run.smithi186.stdout:(109/152): abseil-cpp-20211102.0-3.el9.x86_64.r 429 kB/s | 548 kB 00:01 2024-04-26T22:37:34.801 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T22:37:34.809 INFO:teuthology.orchestra.run.smithi186.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2024-04-26T22:37:34.864 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T22:37:34.911 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T22:37:34.936 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T22:37:34.981 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T22:37:35.009 INFO:teuthology.orchestra.run.smithi186.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-04-26T22:37:35.333 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T22:37:35.445 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:37:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:37:35.579 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:37:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:37:35.683 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:37:35.735 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-26T22:37:35.736 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-26T22:37:35.751 INFO:teuthology.orchestra.run.smithi186.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 739 kB/s | 548 kB 00:00 2024-04-26T22:37:35.839 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T22:37:35.859 INFO:teuthology.orchestra.run.smithi186.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 402 kB/s | 43 kB 00:00 2024-04-26T22:37:35.914 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T22:37:35.942 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:37:35.968 INFO:teuthology.orchestra.run.smithi186.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-26T22:37:36.002 INFO:teuthology.orchestra.run.smithi186.stdout:(115/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 703 kB/s | 839 kB 00:01 2024-04-26T22:37:36.046 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:37:36.069 INFO:teuthology.orchestra.run.smithi186.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-26T22:37:36.076 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:37:36.139 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:37:36.160 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:37:36.183 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:37:36.269 INFO:teuthology.orchestra.run.smithi186.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 644 kB/s | 172 kB 00:00 2024-04-26T22:37:36.377 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:37:36.410 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:37:36.521 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T22:37:36.569 INFO:teuthology.orchestra.run.smithi186.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 717 kB/s | 358 kB 00:00 2024-04-26T22:37:36.630 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T22:37:36.668 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T22:37:36.678 INFO:teuthology.orchestra.run.smithi186.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-26T22:37:36.703 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T22:37:36.703 INFO:teuthology.orchestra.run.smithi186.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 699 kB/s | 303 kB 00:00 2024-04-26T22:37:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T22:37:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T22:37:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T22:37:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T22:37:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T22:37:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T22:37:37.103 INFO:teuthology.orchestra.run.smithi186.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 483 kB/s | 206 kB 00:00 2024-04-26T22:37:37.122 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T22:37:37.150 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T22:37:37.194 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T22:37:37.272 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T22:37:37.321 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T22:37:37.365 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T22:37:37.393 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T22:37:37.404 INFO:teuthology.orchestra.run.smithi186.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 482 kB/s | 144 kB 00:00 2024-04-26T22:37:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T22:37:37.512 INFO:teuthology.orchestra.run.smithi186.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-26T22:37:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T22:37:37.595 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T22:37:37.621 INFO:teuthology.orchestra.run.smithi186.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-04-26T22:37:37.721 INFO:teuthology.orchestra.run.smithi186.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-26T22:37:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T22:37:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T22:37:37.821 INFO:teuthology.orchestra.run.smithi186.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-26T22:37:37.922 INFO:teuthology.orchestra.run.smithi186.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-04-26T22:37:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T22:37:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T22:37:38.469 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T22:37:38.512 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T22:37:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T22:37:38.583 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T22:37:38.745 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T22:37:38.802 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T22:37:38.875 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T22:37:38.950 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T22:37:38.986 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T22:37:39.023 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T22:37:39.070 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T22:37:39.105 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T22:37:39.149 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T22:37:39.191 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T22:37:39.214 INFO:teuthology.orchestra.run.smithi186.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 809 kB/s | 1.0 MB 00:01 2024-04-26T22:37:39.331 INFO:teuthology.orchestra.run.smithi186.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 408 kB/s | 46 kB 00:00 2024-04-26T22:37:39.334 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T22:37:39.380 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T22:37:39.409 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T22:37:39.414 INFO:teuthology.orchestra.run.smithi186.stdout:(130/152): python3-grpcio-1.46.7-10.el9.x86_64. 770 kB/s | 2.0 MB 00:02 2024-04-26T22:37:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T22:37:39.523 INFO:teuthology.orchestra.run.smithi186.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 410 kB/s | 79 kB 00:00 2024-04-26T22:37:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T22:37:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T22:37:39.623 INFO:teuthology.orchestra.run.smithi186.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 277 kB/s | 58 kB 00:00 2024-04-26T22:37:39.624 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T22:37:39.657 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T22:37:39.682 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T22:37:39.723 INFO:teuthology.orchestra.run.smithi186.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-26T22:37:39.761 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T22:37:39.897 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T22:37:39.898 INFO:teuthology.orchestra.run.smithi186.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 514 kB/s | 90 kB 00:00 2024-04-26T22:37:39.939 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T22:37:39.957 INFO:teuthology.orchestra.run.smithi186.stdout:(135/152): libarrow-9.0.0-11.el9.x86_64.rpm 757 kB/s | 4.4 MB 00:05 2024-04-26T22:37:39.968 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T22:37:39.999 INFO:teuthology.orchestra.run.smithi186.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 572 kB/s | 272 kB 00:00 2024-04-26T22:37:40.002 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T22:37:40.016 INFO:teuthology.orchestra.run.smithi186.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-04-26T22:37:40.041 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T22:37:40.049 INFO:teuthology.orchestra.run.smithi186.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-04-26T22:37:40.073 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T22:37:40.102 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T22:37:40.137 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T22:37:40.150 INFO:teuthology.orchestra.run.smithi186.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-26T22:37:40.170 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T22:37:40.191 INFO:teuthology.orchestra.run.smithi186.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-04-26T22:37:40.213 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T22:37:40.246 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T22:37:40.250 INFO:teuthology.orchestra.run.smithi186.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-04-26T22:37:40.276 INFO:teuthology.orchestra.run.smithi186.stdout:(142/152): python3-routes-2.5.1-5.el9.noarch.rp 683 kB/s | 188 kB 00:00 2024-04-26T22:37:40.378 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T22:37:40.428 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T22:37:40.442 INFO:teuthology.orchestra.run.smithi186.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 468 kB/s | 90 kB 00:00 2024-04-26T22:37:40.458 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T22:37:40.506 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T22:37:40.551 INFO:teuthology.orchestra.run.smithi186.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-26T22:37:40.576 INFO:teuthology.orchestra.run.smithi186.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 597 kB/s | 230 kB 00:00 2024-04-26T22:37:40.618 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T22:37:40.677 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T22:37:40.860 INFO:teuthology.orchestra.run.smithi186.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 622 kB/s | 192 kB 00:00 2024-04-26T22:37:40.968 INFO:teuthology.orchestra.run.smithi186.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 617 kB/s | 427 kB 00:00 2024-04-26T22:37:41.411 INFO:teuthology.orchestra.run.smithi186.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 6.2 MB/s | 3.4 MB 00:00 2024-04-26T22:37:41.439 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T22:37:41.481 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T22:37:41.515 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T22:37:41.552 INFO:teuthology.orchestra.run.smithi186.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 5.4 MB/s | 3.1 MB 00:00 2024-04-26T22:37:41.619 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T22:37:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T22:37:41.763 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T22:37:41.800 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T22:37:41.845 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-26T22:37:41.878 INFO:teuthology.orchestra.run.smithi186.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 2.9 MB/s | 1.3 MB 00:00 2024-04-26T22:37:41.932 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T22:37:41.971 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T22:37:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T22:37:42.023 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T22:37:42.101 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T22:37:42.103 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T22:37:42.217 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T22:37:42.250 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T22:37:42.261 INFO:teuthology.orchestra.run.smithi186.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 2.4 MB 00:00 2024-04-26T22:37:42.295 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T22:37:42.570 INFO:teuthology.orchestra.run.smithi186.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 813 kB/s | 1.6 MB 00:01 2024-04-26T22:37:42.577 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:37:42.577 INFO:teuthology.orchestra.run.smithi186.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-26T22:37:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T22:37:42.751 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:37:42.784 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T22:37:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:37:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T22:37:42.911 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:37:43.003 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:37:43.041 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:37:43.199 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T22:37:43.255 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T22:37:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:37:43.387 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:37:43.425 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:37:43.487 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:37:43.509 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:37:43.533 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:37:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:37:43.708 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-26T22:37:43.726 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:37:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T22:37:43.851 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-26T22:37:43.851 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-26T22:37:43.933 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T22:37:43.967 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T22:37:44.002 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T22:37:44.078 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T22:37:44.118 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T22:37:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T22:37:44.194 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T22:37:44.203 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T22:37:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T22:37:44.277 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T22:37:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T22:37:44.314 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T22:37:44.346 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T22:37:44.354 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T22:37:44.383 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T22:37:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T22:37:44.497 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T22:37:44.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T22:37:44.565 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T22:37:44.593 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T22:37:44.666 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T22:37:44.675 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T22:37:44.708 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T22:37:44.709 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:37:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:37:44.786 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T22:37:44.814 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T22:37:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T22:37:44.932 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T22:37:44.987 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T22:37:45.356 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T22:37:45.536 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:37:45.549 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T22:37:45.608 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:37:45.622 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-26T22:37:45.622 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-26T22:37:45.638 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T22:37:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T22:37:45.677 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T22:37:45.720 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T22:37:45.750 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T22:37:45.792 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T22:37:45.963 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T22:37:46.020 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T22:37:46.085 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T22:37:46.166 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T22:37:46.383 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T22:37:46.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T22:37:46.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T22:37:46.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T22:37:46.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T22:37:46.409 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T22:37:46.553 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T22:37:46.598 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T22:37:46.628 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T22:37:46.670 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T22:37:46.768 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T22:37:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T22:37:46.844 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T22:37:46.877 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T22:37:46.901 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T22:37:46.980 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T22:37:47.109 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T22:37:47.142 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T22:37:47.170 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T22:37:47.205 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T22:37:47.245 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T22:37:47.284 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T22:37:47.314 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T22:37:47.348 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T22:37:47.382 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T22:37:47.424 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T22:37:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T22:37:47.550 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T22:37:47.598 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T22:37:47.628 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T22:37:47.676 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T22:37:47.788 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T22:37:47.848 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T22:37:48.005 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:37:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:37:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:37:48.620 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T22:37:48.654 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:37:48.670 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T22:37:48.703 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T22:37:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:37:48.750 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T22:37:48.750 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T22:37:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:37:48.780 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:37:48.806 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T22:37:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T22:37:48.953 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T22:37:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T22:37:49.191 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T22:37:49.219 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T22:37:49.300 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T22:37:49.993 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T22:37:50.106 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T22:37:51.424 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T22:37:51.509 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T22:37:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T22:37:51.563 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-26T22:37:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T22:37:51.649 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T22:37:51.771 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T22:37:51.899 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T22:37:51.941 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:37:52.002 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T22:37:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:37:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T22:37:52.050 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T22:37:52.075 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T22:37:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T22:37:52.119 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T22:37:52.448 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T22:37:52.550 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:37:52.589 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T22:37:52.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:37:52.671 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:37:52.682 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:37:52.701 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:37:52.721 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T22:37:52.801 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:37:52.831 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:37:52.979 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T22:37:53.028 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T22:37:53.055 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:37:53.150 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:37:53.173 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:37:53.219 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:37:53.240 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:37:53.264 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:37:53.457 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:37:53.490 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:37:53.598 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T22:37:53.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T22:37:53.739 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T22:37:53.782 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T22:37:53.847 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T22:37:53.890 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T22:37:53.933 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T22:37:53.965 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T22:37:53.994 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T22:37:54.054 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T22:37:54.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T22:37:54.138 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T22:37:54.173 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T22:37:54.249 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T22:37:54.284 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T22:37:54.327 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T22:37:54.356 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T22:37:54.426 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T22:37:54.479 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T22:37:54.562 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T22:37:54.704 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T22:37:54.756 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T22:37:55.231 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:37:55.268 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:37:55.273 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T22:37:55.368 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T22:37:55.397 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T22:37:55.443 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T22:37:55.479 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T22:37:55.520 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T22:37:55.684 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T22:37:55.739 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T22:37:55.794 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T22:37:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:37:55.867 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T22:37:55.888 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:37:55.898 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T22:37:55.935 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T22:37:55.970 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:37:55.971 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T22:37:55.971 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T22:37:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:37:55.981 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T22:37:55.989 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:37:56.017 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T22:37:56.061 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T22:37:56.103 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T22:37:56.238 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T22:37:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T22:37:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T22:37:56.363 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T22:37:56.457 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T22:37:56.488 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T22:37:56.528 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T22:37:56.620 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T22:37:56.645 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T22:37:56.720 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T22:37:56.847 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T22:37:56.886 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T22:37:56.914 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T22:37:57.092 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T22:37:57.229 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T22:37:57.260 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T22:37:57.290 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T22:37:57.324 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T22:37:57.357 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T22:37:57.399 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T22:37:57.434 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T22:37:57.512 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T22:37:57.555 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T22:37:57.587 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T22:37:57.634 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T22:37:57.732 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T22:37:57.780 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T22:37:58.500 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T22:37:58.560 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T22:37:58.594 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T22:37:58.693 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T22:37:58.743 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T22:37:58.830 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T22:37:58.904 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T22:37:59.083 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T22:37:59.110 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T22:37:59.188 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T22:37:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T22:37:59.957 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T22:38:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:38:00.629 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-04-26T22:38:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:00.826 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:38:00.842 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:38:00.843 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:00.843 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T22:38:00.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:01.195 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T22:38:01.271 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T22:38:01.291 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T22:38:01.316 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T22:38:01.627 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T22:38:01.679 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:38:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:38:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T22:38:01.836 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T22:38:02.328 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T22:38:02.365 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:38:02.429 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:38:02.435 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:38:02.456 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:38:02.456 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:02.456 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T22:38:02.456 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:38:02.456 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:38:02.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:02.458 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T22:38:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:38:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:38:02.712 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:02.712 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T22:38:02.712 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:38:02.712 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:38:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:38:03.204 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:38:03.204 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:03.204 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T22:38:03.204 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:38:03.204 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:38:03.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:03.320 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T22:38:03.378 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T22:38:03.441 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T22:38:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T22:38:03.682 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T22:38:03.796 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T22:38:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:38:04.042 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:38:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T22:38:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T22:38:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T22:38:04.594 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:38:04.623 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:38:04.655 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T22:38:04.716 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:38:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T22:38:04.745 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:38:04.754 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:38:05.295 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:38:05.341 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:38:05.443 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:38:05.443 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T22:38:05.443 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T22:38:05.443 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:05.475 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:38:05.839 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:38:05.880 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:38:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:38:06.923 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:38:07.031 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:38:07.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T22:38:07.225 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:38:07.248 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:38:07.249 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:07.249 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T22:38:07.249 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:38:07.249 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:38:07.249 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:07.300 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:38:07.314 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:38:07.424 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T22:38:07.467 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T22:38:08.239 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:38:08.239 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-26T22:38:08.239 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-26T22:38:08.240 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-26T22:38:08.240 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-26T22:38:08.240 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-26T22:38:08.240 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-26T22:38:08.240 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-26T22:38:08.240 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:08.414 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:38:08.431 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:38:08.431 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:08.431 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T22:38:08.431 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:08.440 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T22:38:08.469 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:38:08.493 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:38:08.493 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:08.493 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T22:38:08.494 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:38:08.494 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:38:08.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:08.545 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:38:08.568 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:38:08.568 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:08.568 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T22:38:08.568 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:08.612 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:38:08.636 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:38:08.636 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:08.636 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T22:38:08.636 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:38:08.636 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:38:08.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:08.955 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:38:08.978 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:38:08.978 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:08.978 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T22:38:08.978 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:38:08.978 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:38:08.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:09.967 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:38:09.995 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:38:09.995 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:09.995 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T22:38:09.995 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:38:09.995 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:38:09.995 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:10.235 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:38:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:38:10.259 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:10.259 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T22:38:10.259 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:38:10.259 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:38:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:38:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:38:10.751 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:10.752 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T22:38:10.752 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:38:10.752 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:38:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:10.867 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T22:38:10.926 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T22:38:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T22:38:11.176 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T22:38:11.221 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T22:38:11.303 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T22:38:11.392 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:38:11.583 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:38:11.621 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T22:38:11.653 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T22:38:11.680 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T22:38:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:38:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:38:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T22:38:12.208 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T22:38:12.236 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:38:13.332 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:38:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:38:13.826 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T22:38:13.867 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T22:38:13.909 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T22:38:13.940 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T22:38:13.967 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T22:38:14.160 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T22:38:14.205 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T22:38:14.234 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T22:38:14.380 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:38:14.405 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:38:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:38:14.682 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T22:38:14.707 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:38:14.732 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:38:14.732 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:14.732 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T22:38:14.732 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:38:14.732 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:38:14.732 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:14.783 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:38:14.794 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:38:14.871 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:14.887 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T22:38:14.916 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:14.936 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:14.941 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T22:38:15.849 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T22:38:15.917 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T22:38:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T22:38:15.944 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:38:15.968 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:38:15.968 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:15.968 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T22:38:15.968 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:38:15.968 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:38:15.968 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:15.970 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T22:38:16.020 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:38:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:38:16.043 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:16.044 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T22:38:16.044 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:16.063 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T22:38:16.095 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:38:16.099 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T22:38:16.099 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:38:16.119 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:38:16.119 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:16.119 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T22:38:16.120 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:38:16.120 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:38:16.120 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:16.183 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:38:16.216 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:38:16.216 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:38:16.303 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:38:16.376 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:38:16.462 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:38:16.462 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:16.462 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T22:38:16.462 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:38:16.462 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:38:16.462 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:17.088 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:38:17.088 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-26T22:38:17.088 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-26T22:38:17.089 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-26T22:38:17.089 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-26T22:38:17.089 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-26T22:38:17.089 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-26T22:38:17.089 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-26T22:38:17.089 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:17.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:38:17.277 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:38:17.278 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:17.278 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T22:38:17.278 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:18.764 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:38:18.790 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:38:18.790 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:18.790 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T22:38:18.791 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:38:18.791 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:38:18.791 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:19.015 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:38:19.037 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:38:19.037 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:19.038 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T22:38:19.038 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:38:19.038 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:38:19.038 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:19.474 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:38:19.497 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:38:19.497 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:19.497 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T22:38:19.497 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:38:19.497 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:38:19.497 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:19.596 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T22:38:19.652 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T22:38:19.716 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T22:38:19.901 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T22:38:19.950 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T22:38:20.027 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T22:38:20.123 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:38:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:38:20.316 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T22:38:20.348 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T22:38:20.376 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T22:38:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:38:20.849 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:38:20.881 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T22:38:20.951 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T22:38:20.981 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:38:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T22:38:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T22:38:21.418 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T22:38:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T22:38:21.476 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T22:38:21.670 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T22:38:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T22:38:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T22:38:22.011 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:38:22.049 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:38:22.260 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:22.260 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T22:38:22.260 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T22:38:22.260 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T22:38:22.260 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:38:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T22:38:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T22:38:22.263 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T22:38:22.264 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T22:38:22.265 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T22:38:22.266 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T22:38:22.267 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T22:38:22.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T22:38:22.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T22:38:22.270 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T22:38:22.271 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T22:38:22.272 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T22:38:22.273 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:38:22.274 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T22:38:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:23.007 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:38:23.066 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:38:23.166 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:38:23.264 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T22:38:23.327 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T22:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T22:38:23.359 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:38:23.383 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:38:23.384 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:23.384 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T22:38:23.384 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:38:23.384 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:38:23.384 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:23.400 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T22:38:23.435 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:38:23.446 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:38:23.484 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T22:38:23.512 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T22:38:23.513 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:38:23.543 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T22:38:23.591 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T22:38:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:38:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:38:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:38:23.746 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:38:23.814 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-04-26T22:38:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T22:38:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T22:38:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T22:38:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T22:38:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T22:38:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T22:38:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T22:38:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T22:38:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T22:38:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T22:38:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:38:24.352 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-04-26T22:38:24.512 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T22:38:24.546 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:38:24.569 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:38:24.569 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:24.569 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T22:38:24.569 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:38:24.569 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:38:24.569 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:24.614 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:38:24.636 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:38:24.636 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:24.636 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T22:38:24.636 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:24.680 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:38:24.703 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:38:24.703 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:24.703 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T22:38:24.703 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:38:24.703 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:38:24.703 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:24.741 DEBUG:teuthology.parallel:result is None 2024-04-26T22:38:25.006 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:38:25.029 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:38:25.029 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:38:25.029 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T22:38:25.029 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:38:25.029 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:38:25.029 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:29.618 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T22:38:29.639 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:29.639 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T22:38:29.639 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T22:38:29.639 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T22:38:29.639 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T22:38:29.640 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T22:38:29.641 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T22:38:29.654 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T22:38:29.655 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T22:38:29.656 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T22:38:29.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T22:38:29.658 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T22:38:29.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T22:38:29.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T22:38:29.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T22:38:29.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T22:38:29.663 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T22:38:29.663 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T22:38:29.663 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T22:38:29.663 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:38:29.663 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T22:38:29.663 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T22:38:29.700 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T22:38:29.732 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T22:38:29.759 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T22:38:29.943 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T22:38:29.980 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T22:38:30.010 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T22:38:30.631 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:30.675 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:30.695 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:38:31.543 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T22:38:31.629 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T22:38:31.670 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T22:38:31.752 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T22:38:31.783 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T22:38:31.783 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:38:31.845 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T22:38:31.846 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.847 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T22:38:31.848 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.849 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T22:38:31.850 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T22:38:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T22:38:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T22:38:31.853 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T22:38:31.854 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T22:38:31.855 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T22:38:31.856 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T22:38:31.857 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T22:38:31.858 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T22:38:31.858 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T22:38:31.858 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-26T22:38:31.867 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:38:31.900 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:38:31.900 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:38:31.997 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:38:32.065 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:32.223 DEBUG:teuthology.parallel:result is None 2024-04-26T22:38:37.859 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:37.859 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T22:38:37.859 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T22:38:37.859 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:38:37.860 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T22:38:37.861 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T22:38:37.862 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T22:38:37.862 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T22:38:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T22:38:37.877 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T22:38:37.878 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T22:38:37.879 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T22:38:37.880 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T22:38:37.881 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T22:38:37.882 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T22:38:37.883 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T22:38:37.884 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T22:38:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T22:38:37.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T22:38:37.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T22:38:37.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T22:38:37.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T22:38:37.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T22:38:37.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:38:37.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T22:38:39.924 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T22:38:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.926 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:38:39.927 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.928 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T22:38:39.929 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T22:38:39.930 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T22:38:39.931 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T22:38:39.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T22:38:39.933 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T22:38:39.934 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T22:38:39.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T22:38:39.936 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T22:38:39.937 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T22:38:39.937 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:38:39.937 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-26T22:38:40.337 DEBUG:teuthology.parallel:result is None 2024-04-26T22:38:40.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:38:40.481 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T22:38:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T22:38:40.509 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T22:38:40.509 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T22:38:40.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:38:40.653 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T22:38:40.680 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T22:38:40.680 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T22:38:40.680 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T22:38:40.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:38:40.821 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T22:38:40.842 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T22:38:40.842 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T22:38:40.842 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T22:38:40.844 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T22:38:40.845 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:38:40.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T22:38:40.877 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:38:40.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T22:38:40.909 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:38:40.909 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T22:38:40.937 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T22:38:40.937 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:38:40.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T22:38:40.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T22:38:41.036 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:38:41.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T22:38:41.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T22:38:41.135 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:38:41.135 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T22:38:41.158 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T22:38:41.222 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T22:38:41.222 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:38:41.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T22:38:41.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T22:38:41.319 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:38:41.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T22:38:41.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T22:38:41.419 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:38:41.419 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T22:38:41.441 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T22:38:41.506 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T22:38:41.506 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:38:41.506 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T22:38:41.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T22:38:41.596 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:38:41.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T22:38:41.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T22:38:41.685 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:38:41.685 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T22:38:41.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T22:38:41.772 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T22:38:41.879 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-26T22:38:41.880 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:38:41.880 INFO:tasks.cephadm:Cluster fsid is bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:38:41.880 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T22:38:41.881 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.1', 'mon.b': '172.21.15.38', 'mon.c': '172.21.15.186'} 2024-04-26T22:38:41.881 INFO:tasks.cephadm:First mon is mon.a on smithi001 2024-04-26T22:38:41.881 INFO:tasks.cephadm:First mgr is x 2024-04-26T22:38:41.881 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T22:38:41.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo hostname $(hostname -s) 2024-04-26T22:38:41.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-26T22:38:41.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-04-26T22:38:41.955 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T22:38:41.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:38:42.175 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T22:38:42.313 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:38:42.314 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T22:38:42.315 DEBUG:teuthology.orchestra.run.smithi001:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:42.773 INFO:teuthology.orchestra.run.smithi001.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 22:38 /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:42.773 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:43.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 22:38 /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:43.412 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:43.843 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 22:38 /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:43.843 DEBUG:teuthology.orchestra.run.smithi001:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:43.864 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:43.883 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T22:38:44.028 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T22:38:44.028 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T22:38:44.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T22:38:44.034 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T22:38:45.604 INFO:teuthology.orchestra.run.smithi001.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T22:38:45.644 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T22:38:45.859 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T22:39:14.292 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-26T22:39:14.292 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T22:39:14.292 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T22:39:14.293 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-26T22:39:14.293 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T22:39:14.293 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-26T22:39:14.293 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-26T22:39:16.624 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T22:39:17.670 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:39:17.671 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T22:39:17.671 INFO:teuthology.orchestra.run.smithi001.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T22:39:17.671 INFO:teuthology.orchestra.run.smithi001.stdout: "repo_digests": [ 2024-04-26T22:39:17.671 INFO:teuthology.orchestra.run.smithi001.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T22:39:17.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T22:39:17.671 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:39:17.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph 2024-04-26T22:39:17.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-26T22:39:17.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-04-26T22:39:17.809 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /etc/ceph 2024-04-26T22:39:17.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-26T22:39:17.863 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-04-26T22:39:17.889 INFO:tasks.cephadm:Writing seed config... 2024-04-26T22:39:17.890 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T22:39:17.890 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T22:39:17.890 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T22:39:17.890 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-26T22:39:17.891 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T22:39:17.892 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T22:39:17.893 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-26T22:39:17.894 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:39:17.894 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T22:39:17.910 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = bb8b501a-041d-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T22:39:17.910 DEBUG:teuthology.orchestra.run.smithi001:mon.a> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T22:39:17.953 DEBUG:teuthology.orchestra.run.smithi001:mgr.x> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T22:39:17.996 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T22:39:17.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.1 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T22:39:18.191 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:39:18.191 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', 'bb8b501a-041d-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.1', '--skip-admin-label'] 2024-04-26T22:39:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T22:39:18.192 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying podman|docker is present... 2024-04-26T22:39:18.217 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T22:39:18.217 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying lvm2 is present... 2024-04-26T22:39:18.217 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying time synchronization is in place... 2024-04-26T22:39:18.224 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T22:39:18.224 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T22:39:18.230 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T22:39:18.230 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T22:39:18.237 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-04-26T22:39:18.243 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-04-26T22:39:18.243 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-04-26T22:39:18.243 INFO:teuthology.orchestra.run.smithi001.stdout:Repeating the final host check... 2024-04-26T22:39:18.267 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T22:39:18.267 INFO:teuthology.orchestra.run.smithi001.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T22:39:18.267 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl is present 2024-04-26T22:39:18.268 INFO:teuthology.orchestra.run.smithi001.stdout:lvcreate is present 2024-04-26T22:39:18.274 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T22:39:18.275 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T22:39:18.279 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T22:39:18.280 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T22:39:18.286 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-04-26T22:39:18.292 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-04-26T22:39:18.292 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-04-26T22:39:18.292 INFO:teuthology.orchestra.run.smithi001.stdout:Host looks OK 2024-04-26T22:39:18.293 INFO:teuthology.orchestra.run.smithi001.stdout:Cluster fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:18.293 INFO:teuthology.orchestra.run.smithi001.stdout:Acquiring lock 139688388329424 on /run/cephadm/bb8b501a-041d-11ef-bc93-c7b262605968.lock 2024-04-26T22:39:18.293 INFO:teuthology.orchestra.run.smithi001.stdout:Lock 139688388329424 acquired on /run/cephadm/bb8b501a-041d-11ef-bc93-c7b262605968.lock 2024-04-26T22:39:18.293 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 3300 ... 2024-04-26T22:39:18.293 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 6789 ... 2024-04-26T22:39:18.294 INFO:teuthology.orchestra.run.smithi001.stdout:Base mon IP(s) is [172.21.15.1:3300, 172.21.15.1:6789], mon addrv is [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-04-26T22:39:18.297 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.1 metric 100 2024-04-26T22:39:18.297 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.1 metric 100 2024-04-26T22:39:18.299 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T22:39:18.299 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T22:39:18.299 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T22:39:18.302 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T22:39:18.302 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T22:39:18.302 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T22:39:18.302 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T22:39:18.302 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-04-26T22:39:18.302 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T22:39:18.303 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-04-26T22:39:18.303 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-04-26T22:39:18.303 INFO:teuthology.orchestra.run.smithi001.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T22:39:18.303 INFO:teuthology.orchestra.run.smithi001.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T22:39:18.304 INFO:teuthology.orchestra.run.smithi001.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T22:39:18.798 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T22:39:18.798 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T22:39:18.798 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T22:39:18.798 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T22:39:18.799 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T22:39:18.799 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T22:39:18.799 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T22:39:19.735 INFO:teuthology.orchestra.run.smithi001.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T22:39:19.735 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T22:39:19.735 INFO:teuthology.orchestra.run.smithi001.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T22:39:20.704 INFO:teuthology.orchestra.run.smithi001.stdout:stat: stdout 167 167 2024-04-26T22:39:20.704 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial keys... 2024-04-26T22:39:21.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQAZLSxmAr5HABAAj5mfmRbeGCotC21+1JAdGQ== 2024-04-26T22:39:22.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQAZLSxmySbkOBAAznj3Zlq3bNLGqrwlqJvdNA== 2024-04-26T22:39:23.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQAaLSxmCSDwNBAAwplcRjnE8bRnwReMg4gQiA== 2024-04-26T22:39:23.550 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial monmap... 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool for a [v2:172.21.15.1:3300,v1:172.21.15.1:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: set fsid to bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:24.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T22:39:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:39:24.416 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mon... 2024-04-26T22:39:25.286 INFO:teuthology.orchestra.run.smithi001.stdout:create mon.a on 2024-04-26T22:39:25.518 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T22:39:25.690 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T22:39:25.886 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-bb8b501a-041d-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-bb8b501a-041d-11ef-bc93-c7b262605968.target. 2024-04-26T22:39:25.886 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-bb8b501a-041d-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-bb8b501a-041d-11ef-bc93-c7b262605968.target. 2024-04-26T22:39:26.146 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a 2024-04-26T22:39:26.146 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T22:39:26.330 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bb8b501a-041d-11ef-bc93-c7b262605968.target.wants/ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-bb8b501a-041d-11ef-bc93-c7b262605968@.service. 2024-04-26T22:39:26.872 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T22:39:26.872 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T22:39:26.880 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T22:39:26.880 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T22:39:26.880 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T22:39:26.880 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T22:39:26.881 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon to start... 2024-04-26T22:39:26.881 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon... 2024-04-26T22:39:26.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:26 smithi001 systemd[1]: Started Ceph mon.a for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:39:27.974 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T22:39:27.974 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: monmap epoch 1 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: last_changed 2024-04-26T22:39:23.873268+0000 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: min_mon_release 19 (squid) 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: election_strategy: 1 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: fsmap 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: mgrmap e1: no daemons active 2024-04-26T22:39:27.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:27 smithi001 ceph-mon[29065]: from='client.? 172.21.15.1:0/4002066123' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout id: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout services: 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.556302s) 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout data: 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.310+0000 7f2006f48700 1 Processor -- start 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2006f48700 1 -- start start 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2006f48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f20001080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2006f48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2000108680 con 0x7f2000107c90 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2004ce4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f20001080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2004ce4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f20001080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42368/0 (socket says 172.21.15.1:42368) 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2004ce4700 1 -- 172.21.15.1:0/3095648771 learned_addr learned my addr 172.21.15.1:0/3095648771 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.311+0000 7f2004ce4700 1 -- 172.21.15.1:0/3095648771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2000108e90 con 0x7f2000107c90 2024-04-26T22:39:27.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2004ce4700 1 --2- 172.21.15.1:0/3095648771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f20001080b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40098a0 tx=0x7f1ff4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570a4d7eeea3c397 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f1fff7fe700 1 -- 172.21.15.1:0/3095648771 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ff4004160 con 0x7f2000107c90 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f1fff7fe700 1 -- 172.21.15.1:0/3095648771 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff4004300 con 0x7f2000107c90 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f1fff7fe700 1 -- 172.21.15.1:0/3095648771 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ff40036a0 con 0x7f2000107c90 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 -- 172.21.15.1:0/3095648771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 msgr2=0x7f20001080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 --2- 172.21.15.1:0/3095648771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f20001080b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40098a0 tx=0x7f1ff4009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 -- 172.21.15.1:0/3095648771 shutdown_connections 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 --2- 172.21.15.1:0/3095648771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f20001080b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 -- 172.21.15.1:0/3095648771 >> 172.21.15.1:0/3095648771 conn(0x7f20001031f0 msgr2=0x7f2000105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 -- 172.21.15.1:0/3095648771 shutdown_connections 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.312+0000 7f2006f48700 1 -- 172.21.15.1:0/3095648771 wait complete. 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2006f48700 1 Processor -- start 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2006f48700 1 -- start start 2024-04-26T22:39:27.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2006f48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f200019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2006f48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20001093c0 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2004ce4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f200019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2004ce4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f200019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42374/0 (socket says 172.21.15.1:42374) 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.313+0000 7f2004ce4700 1 -- 172.21.15.1:0/4002066123 learned_addr learned my addr 172.21.15.1:0/4002066123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f2004ce4700 1 -- 172.21.15.1:0/4002066123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200019ca00 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f2004ce4700 1 --2- 172.21.15.1:0/4002066123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f200019c4c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4000c00 tx=0x7f1ff4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ff4003db0 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff4011630 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200019cc90 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ff4011950 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200019f980 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1ff401a460 con 0x7f2000107c90 2024-04-26T22:39:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.314+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1ff401b780 con 0x7f2000107c90 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.315+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2000064a80 con 0x7f2000107c90 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.318+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f1ff4031020 con 0x7f2000107c90 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.391+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f200002e790 con 0x7f2000107c90 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.391+0000 7f1ffdffb700 1 -- 172.21.15.1:0/4002066123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f1ff401ba20 con 0x7f2000107c90 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 msgr2=0x7f200019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 --2- 172.21.15.1:0/4002066123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f200019c4c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4000c00 tx=0x7f1ff4004750 comp rx=0 tx=0).stop 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 shutdown_connections 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 --2- 172.21.15.1:0/4002066123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2000107c90 0x7f200019c4c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 >> 172.21.15.1:0/4002066123 conn(0x7f20001031f0 msgr2=0x7f2000193050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 shutdown_connections 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:27.392+0000 7f2006f48700 1 -- 172.21.15.1:0/4002066123 wait complete. 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:mon is available 2024-04-26T22:39:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T22:39:29.024 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:28 smithi001 ceph-mon[29065]: from='client.? 172.21.15.1:0/3433091107' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T22:39:29.024 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:28 smithi001 ceph-mon[29065]: from='client.? 172.21.15.1:0/3433091107' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T22:39:29.033 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T22:39:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b502d6700 1 Processor -- start 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b502d6700 1 -- start start 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b502d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b481091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b502d6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4b481097c0 con 0x7f4b48108dd0 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b4e072700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b481091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b4e072700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b481091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42388/0 (socket says 172.21.15.1:42388) 2024-04-26T22:39:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.389+0000 7f4b4e072700 1 -- 172.21.15.1:0/2005131692 learned_addr learned my addr 172.21.15.1:0/2005131692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.390+0000 7f4b4e072700 1 -- 172.21.15.1:0/2005131692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b48109f80 con 0x7f4b48108dd0 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.390+0000 7f4b4e072700 1 --2- 172.21.15.1:0/2005131692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b481091f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4b440098a0 tx=0x7f4b44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faee264c66740b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.390+0000 7f4b4d070700 1 -- 172.21.15.1:0/2005131692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b44004160 con 0x7f4b48108dd0 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.390+0000 7f4b4d070700 1 -- 172.21.15.1:0/2005131692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b44004300 con 0x7f4b48108dd0 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.390+0000 7f4b4d070700 1 -- 172.21.15.1:0/2005131692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b440036a0 con 0x7f4b48108dd0 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 -- 172.21.15.1:0/2005131692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 msgr2=0x7f4b481091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 --2- 172.21.15.1:0/2005131692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b481091f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4b440098a0 tx=0x7f4b44009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 -- 172.21.15.1:0/2005131692 shutdown_connections 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 --2- 172.21.15.1:0/2005131692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b481091f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 -- 172.21.15.1:0/2005131692 >> 172.21.15.1:0/2005131692 conn(0x7f4b48104370 msgr2=0x7f4b481067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 -- 172.21.15.1:0/2005131692 shutdown_connections 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 -- 172.21.15.1:0/2005131692 wait complete. 2024-04-26T22:39:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 Processor -- start 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.391+0000 7f4b502d6700 1 -- start start 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b502d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b4819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b502d6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4810a4b0 con 0x7f4b48108dd0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b4e072700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b4819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b4e072700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b4819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42402/0 (socket says 172.21.15.1:42402) 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b4e072700 1 -- 172.21.15.1:0/3433091107 learned_addr learned my addr 172.21.15.1:0/3433091107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b4e072700 1 -- 172.21.15.1:0/3433091107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4819ca00 con 0x7f4b48108dd0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b4e072700 1 --2- 172.21.15.1:0/3433091107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b4819c4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4b44000c00 tx=0x7f4b44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b44003db0 con 0x7f4b48108dd0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b44011630 con 0x7f4b48108dd0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4819cc90 con 0x7f4b48108dd0 2024-04-26T22:39:29.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b44011950 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.392+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b4819f980 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.393+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b4401a460 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.393+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4b4401b780 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.393+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b48064a80 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.396+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4b4401b980 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.467+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4b4819d170 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.470+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+463 (secure 0 0 0) 0x7f4b4819d170 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b3f7fe700 1 -- 172.21.15.1:0/3433091107 <== mon.0 v2:172.21.15.1:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4b4401a8e0 con 0x7f4b48108dd0 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 msgr2=0x7f4b4819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:29.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 --2- 172.21.15.1:0/3433091107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b4819c4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4b44000c00 tx=0x7f4b44004750 comp rx=0 tx=0).stop 2024-04-26T22:39:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 shutdown_connections 2024-04-26T22:39:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 --2- 172.21.15.1:0/3433091107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b48108dd0 0x7f4b4819c4c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 >> 172.21.15.1:0/3433091107 conn(0x7f4b48104370 msgr2=0x7f4b481930e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 shutdown_connections 2024-04-26T22:39:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:28.471+0000 7f4b502d6700 1 -- 172.21.15.1:0/3433091107 wait complete. 2024-04-26T22:39:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:Generating new minimal ceph.conf... 2024-04-26T22:39:30.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.463+0000 7f9e60d7f700 1 Processor -- start 2024-04-26T22:39:30.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.463+0000 7f9e60d7f700 1 -- start start 2024-04-26T22:39:30.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e60d7f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:30.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e60d7f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c1097c0 con 0x7f9e5c108dd0 2024-04-26T22:39:30.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:30.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42414/0 (socket says 172.21.15.1:42414) 2024-04-26T22:39:30.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5a59c700 1 -- 172.21.15.1:0/3997299424 learned_addr learned my addr 172.21.15.1:0/3997299424 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:30.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5a59c700 1 -- 172.21.15.1:0/3997299424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c109f80 con 0x7f9e5c108dd0 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5a59c700 1 --2- 172.21.15.1:0/3997299424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c1091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009e80 tx=0x7f9e4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3825f4ce273f2c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5959a700 1 -- 172.21.15.1:0/3997299424 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e4c017070 con 0x7f9e5c108dd0 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5959a700 1 -- 172.21.15.1:0/3997299424 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9e4c00b8c0 con 0x7f9e5c108dd0 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.464+0000 7f9e5959a700 1 -- 172.21.15.1:0/3997299424 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e4c004490 con 0x7f9e5c108dd0 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 -- 172.21.15.1:0/3997299424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 msgr2=0x7f9e5c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 --2- 172.21.15.1:0/3997299424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c1091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009e80 tx=0x7f9e4c00b330 comp rx=0 tx=0).stop 2024-04-26T22:39:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 -- 172.21.15.1:0/3997299424 shutdown_connections 2024-04-26T22:39:30.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 --2- 172.21.15.1:0/3997299424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c1091f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:30.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 -- 172.21.15.1:0/3997299424 >> 172.21.15.1:0/3997299424 conn(0x7f9e5c104370 msgr2=0x7f9e5c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:30.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 -- 172.21.15.1:0/3997299424 shutdown_connections 2024-04-26T22:39:30.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.465+0000 7f9e60d7f700 1 -- 172.21.15.1:0/3997299424 wait complete. 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e60d7f700 1 Processor -- start 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e60d7f700 1 -- start start 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e60d7f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e60d7f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c10d330 con 0x7f9e5c108dd0 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c19ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42426/0 (socket says 172.21.15.1:42426) 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e5a59c700 1 -- 172.21.15.1:0/293557067 learned_addr learned my addr 172.21.15.1:0/293557067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:30.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.466+0000 7f9e5a59c700 1 -- 172.21.15.1:0/293557067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c19d210 con 0x7f9e5c108dd0 2024-04-26T22:39:30.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e5a59c700 1 --2- 172.21.15.1:0/293557067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c19ccd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00b7b0 tx=0x7f9e4c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:30.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e4c020020 con 0x7f9e5c108dd0 2024-04-26T22:39:30.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9e4c012430 con 0x7f9e5c108dd0 2024-04-26T22:39:30.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5c19d4a0 con 0x7f9e5c108dd0 2024-04-26T22:39:30.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e4c017040 con 0x7f9e5c108dd0 2024-04-26T22:39:30.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5c1a0180 con 0x7f9e5c108dd0 2024-04-26T22:39:30.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9e4c023020 con 0x7f9e5c108dd0 2024-04-26T22:39:30.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.467+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9e4c00f050 con 0x7f9e5c108dd0 2024-04-26T22:39:30.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.468+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e5c064a80 con 0x7f9e5c108dd0 2024-04-26T22:39:30.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.470+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f9e4c02a020 con 0x7f9e5c108dd0 2024-04-26T22:39:30.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.541+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9e5c1a0350 con 0x7f9e5c108dd0 2024-04-26T22:39:30.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.542+0000 7f9e537fe700 1 -- 172.21.15.1:0/293557067 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+173 (secure 0 0 0) 0x7f9e4c00b860 con 0x7f9e5c108dd0 2024-04-26T22:39:30.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 msgr2=0x7f9e5c19ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:30.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 --2- 172.21.15.1:0/293557067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c19ccd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00b7b0 tx=0x7f9e4c00be20 comp rx=0 tx=0).stop 2024-04-26T22:39:30.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 shutdown_connections 2024-04-26T22:39:30.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 --2- 172.21.15.1:0/293557067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5c108dd0 0x7f9e5c19ccd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:30.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 >> 172.21.15.1:0/293557067 conn(0x7f9e5c104370 msgr2=0x7f9e5c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:30.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 shutdown_connections 2024-04-26T22:39:30.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:29.543+0000 7f9e60d7f700 1 -- 172.21.15.1:0/293557067 wait complete. 2024-04-26T22:39:30.098 INFO:teuthology.orchestra.run.smithi001.stdout:Restarting the monitor... 2024-04-26T22:39:30.099 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:29 smithi001 ceph-mon[29065]: from='client.? 172.21.15.1:0/293557067' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:39:30.530 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:30 smithi001 systemd[1]: Stopping Ceph mon.a for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:39:30.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:30 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a[29061]: 2024-04-26T22:39:30.335+0000 7f3617d89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T22:39:30.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:30 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a[29061]: 2024-04-26T22:39:30.335+0000 7f3617d89700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T22:39:30.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:30 smithi001 podman[29309]: 2024-04-26 22:39:30.53017727 +0000 UTC m=+0.279084298 container died 9eacedcb05c3ce11910e7dc3048ce297c96c996f6186045cb209c0b45719de36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T22:39:31.233 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:30 smithi001 podman[29309]: 2024-04-26 22:39:30.870862901 +0000 UTC m=+0.619769917 container cleanup 9eacedcb05c3ce11910e7dc3048ce297c96c996f6186045cb209c0b45719de36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T22:39:31.233 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:30 smithi001 bash[29309]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a 2024-04-26T22:39:31.233 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 podman[29322]: 2024-04-26 22:39:31.019161 +0000 UTC m=+0.490963814 container remove 9eacedcb05c3ce11910e7dc3048ce297c96c996f6186045cb209c0b45719de36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T22:39:31.546 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 systemd[1]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T22:39:31.547 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 systemd[1]: Stopped Ceph mon.a for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:39:31.547 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 systemd[1]: Starting Ceph mon.a for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:39:31.547 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 podman[29415]: 2024-04-26 22:39:31.453139557 +0000 UTC m=+0.098509354 container create f7d0c6f1dd9ebd0dbd04eb3419a7982ca5406d03c372031a8a4b58e242c40674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T22:39:31.547 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 podman[29415]: 2024-04-26 22:39:31.372454365 +0000 UTC m=+0.017824174 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:39:31.768 INFO:teuthology.orchestra.run.smithi001.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T22:39:32.000 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 podman[29415]: 2024-04-26 22:39:31.656079311 +0000 UTC m=+0.301449115 container init f7d0c6f1dd9ebd0dbd04eb3419a7982ca5406d03c372031a8a4b58e242c40674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T22:39:32.000 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 podman[29415]: 2024-04-26 22:39:31.659291019 +0000 UTC m=+0.304660815 container start f7d0c6f1dd9ebd0dbd04eb3419a7982ca5406d03c372031a8a4b58e242c40674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-26T22:39:32.000 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: pidfile_write: ignore empty --pid-file 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: load: jerasure load: lrc 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: RocksDB version: 7.9.2 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Git sha 0 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: DB SUMMARY 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: DB Session ID: RIRECEA9X07TU2P7NKAQ 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: CURRENT file: CURRENT 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104622 ; 2024-04-26T22:39:32.001 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.error_if_exists: 0 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.create_if_missing: 0 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.paranoid_checks: 1 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.env: 0x55b499cdfcc0 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.info_log: 0x55b49aa9bd80 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.statistics: (nil) 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.use_fsync: 0 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_log_file_size: 0 2024-04-26T22:39:32.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.allow_fallocate: 1 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.use_direct_reads: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.db_log_dir: 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.wal_dir: 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T22:39:32.003 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.write_buffer_manager: 0x55b49aa6d0e0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T22:39:32.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.unordered_write: 0 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.row_cache: None 2024-04-26T22:39:32.005 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.wal_filter: None 2024-04-26T22:39:32.006 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T22:39:32.006 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.two_write_queues: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.wal_compression: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.atomic_flush: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.log_readahead_size: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T22:39:32.007 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_background_jobs: 2 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_background_compactions: -1 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_subcompactions: 1 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_open_files: -1 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T22:39:32.008 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_background_flushes: -1 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Compression algorithms supported: 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kZSTD supported: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kXpressCompression supported: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kZlibCompression supported: 1 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kSnappyCompression supported: 1 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kLZ4Compression supported: 1 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: kBZip2Compression supported: 0 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T22:39:32.009 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.merge_operator: 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_filter: None 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b49aa9bf40) 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout: cache_index_and_filter_blocks: 1 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T22:39:32.010 INFO:journalctl@ceph.mon.a.smithi001.stdout: pin_top_level_index_and_filter: 1 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_type: 0 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: data_block_index_type: 0 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_shortening: 1 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: checksum: 4 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: no_block_cache: 0 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache: 0x55b49aa9f1f0 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_name: BinnedLRUCache 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_options: 2024-04-26T22:39:32.011 INFO:journalctl@ceph.mon.a.smithi001.stdout: capacity : 536870912 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: num_shard_bits : 4 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: strict_capacity_limit : 0 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: high_pri_pool_ratio: 0.000 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_compressed: (nil) 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: persistent_cache: (nil) 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_size: 4096 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_size_deviation: 10 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_restart_interval: 16 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_block_restart_interval: 1 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: metadata_block_size: 4096 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: partition_filters: 0 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: use_delta_encoding: 1 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: filter_policy: bloomfilter 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: whole_key_filtering: 1 2024-04-26T22:39:32.012 INFO:journalctl@ceph.mon.a.smithi001.stdout: verify_compression: 0 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: read_amp_bytes_per_bit: 0 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: format_version: 5 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: enable_index_compression: 1 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_align: 0 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: max_auto_readahead_size: 262144 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: prepopulate_block_cache: 0 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: initial_auto_readahead_size: 8192 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression: NoCompression 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T22:39:32.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.num_levels: 7 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T22:39:32.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T22:39:32.015 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T22:39:32.015 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T22:39:32.015 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T22:39:32.015 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T22:39:32.015 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T22:39:32.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T22:39:32.017 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T22:39:32.018 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.inplace_update_support: 0 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T22:39:32.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.bloom_locality: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.max_successive_merges: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.ttl: 2592000 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.enable_blob_files: false 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.min_blob_size: 0 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T22:39:32.020 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1b85db8c-79d6-4656-ba44-b5f8aa21a405 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714171171689484, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714171171691162, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99822, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97708, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14058, "raw_average_key_size": 49, "raw_value_size": 90059, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714171171, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1b85db8c-79d6-4656-ba44-b5f8aa21a405", "db_session_id": "RIRECEA9X07TU2P7NKAQ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714171171691278, "job": 1, "event": "recovery_finished"} 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T22:39:32.021 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b49abbc000 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: DB pointer 0x55b49aba8000 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** DB Stats ** 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T22:39:32.022 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** Compaction Stats [default] ** 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: L0 2/0 99.34 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: Sum 2/0 99.34 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** Compaction Stats [default] ** 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 94.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T22:39:32.023 INFO:journalctl@ceph.mon.a.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative compaction: 0.00 GB write, 20.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval compaction: 0.00 GB write, 20.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: Block cache BinnedLRUCache@0x55b49aa9f1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T22:39:32.024 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: starting mon.a rank 0 at public addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] at bind addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???) e1 preinit fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).mds e1 new map 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).mds e1 print_map 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: e1 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: legacy client fscid: -1 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout: No filesystems configured 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T22:39:32.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: monmap epoch 1 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: last_changed 2024-04-26T22:39:23.873268+0000 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: min_mon_release 19 (squid) 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: election_strategy: 1 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: fsmap 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T22:39:32.026 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 ceph-mon[29435]: mgrmap e1: no daemons active 2024-04-26T22:39:32.027 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 bash[29415]: f7d0c6f1dd9ebd0dbd04eb3419a7982ca5406d03c372031a8a4b58e242c40674 2024-04-26T22:39:32.027 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:31 smithi001 systemd[1]: Started Ceph mon.a for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.179+0000 7f17b1c87700 1 Processor -- start 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.179+0000 7f17b1c87700 1 -- start start 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.179+0000 7f17b1c87700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac10bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.179+0000 7f17b1c87700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac07d6f0 con 0x7f17ac1097e0 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac10bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac10bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43632/0 (socket says 172.21.15.1:43632) 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17ab7fe700 1 -- 172.21.15.1:0/1437402079 learned_addr learned my addr 172.21.15.1:0/1437402079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:32.752 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17ab7fe700 1 -- 172.21.15.1:0/1437402079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ac10c110 con 0x7f17ac1097e0 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17ab7fe700 1 --2- 172.21.15.1:0/1437402079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac10bbd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f179c0098a0 tx=0x7f179c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabcdd1dffc100ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17aa7fc700 1 -- 172.21.15.1:0/1437402079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f179c004160 con 0x7f17ac1097e0 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17aa7fc700 1 -- 172.21.15.1:0/1437402079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f179c004300 con 0x7f17ac1097e0 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.180+0000 7f17aa7fc700 1 -- 172.21.15.1:0/1437402079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f179c011af0 con 0x7f17ac1097e0 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 -- 172.21.15.1:0/1437402079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 msgr2=0x7f17ac10bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 --2- 172.21.15.1:0/1437402079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac10bbd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f179c0098a0 tx=0x7f179c009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 -- 172.21.15.1:0/1437402079 shutdown_connections 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 --2- 172.21.15.1:0/1437402079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac10bbd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 -- 172.21.15.1:0/1437402079 >> 172.21.15.1:0/1437402079 conn(0x7f17ac1031d0 msgr2=0x7f17ac105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 -- 172.21.15.1:0/1437402079 shutdown_connections 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.181+0000 7f17b1c87700 1 -- 172.21.15.1:0/1437402079 wait complete. 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17b1c87700 1 Processor -- start 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17b1c87700 1 -- start start 2024-04-26T22:39:32.753 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17b1c87700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac109060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17b1c87700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac10d0d0 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac109060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac109060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43636/0 (socket says 172.21.15.1:43636) 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.182+0000 7f17ab7fe700 1 -- 172.21.15.1:0/3624587456 learned_addr learned my addr 172.21.15.1:0/3624587456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17ab7fe700 1 -- 172.21.15.1:0/3624587456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ac1095a0 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17ab7fe700 1 --2- 172.21.15.1:0/3624587456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac109060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f179c009fb0 tx=0x7f179c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f179c0036a0 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f179c01a430 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ac107790 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f179c019550 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ac107cd0 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f179c019be0 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.183+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f179c024730 con 0x7f17ac1097e0 2024-04-26T22:39:32.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.184+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ac064a80 con 0x7f17ac1097e0 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.186+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f179c043ec0 con 0x7f17ac1097e0 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.258+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f17ac107f80 con 0x7f17ac1097e0 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.260+0000 7f17a8ff9700 1 -- 172.21.15.1:0/3624587456 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f179c032030 con 0x7f17ac1097e0 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 msgr2=0x7f17ac109060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 --2- 172.21.15.1:0/3624587456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac109060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f179c009fb0 tx=0x7f179c011940 comp rx=0 tx=0).stop 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 shutdown_connections 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 --2- 172.21.15.1:0/3624587456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac1097e0 0x7f17ac109060 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 >> 172.21.15.1:0/3624587456 conn(0x7f17ac1031d0 msgr2=0x7f17ac103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 shutdown_connections 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:32.261+0000 7f17b1c87700 1 -- 172.21.15.1:0/3624587456 wait complete. 2024-04-26T22:39:32.755 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T22:39:32.769 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T22:39:32.769 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mgr... 2024-04-26T22:39:32.769 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T22:39:32.770 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T22:39:33.019 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x 2024-04-26T22:39:33.019 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T22:39:33.176 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bb8b501a-041d-11ef-bc93-c7b262605968.target.wants/ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-bb8b501a-041d-11ef-bc93-c7b262605968@.service. 2024-04-26T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:33 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3624587456' entity='client.admin' 2024-04-26T22:39:33.724 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:33 smithi001 podman[29653]: 2024-04-26 22:39:33.652014652 +0000 UTC m=+0.356863064 container start 2b3a40c98d9a07fb5f9f9b8d295ead2c4fd58d3f1f17451ca98dd73e0bbfeaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T22:39:33.724 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:33 smithi001 bash[29653]: 2b3a40c98d9a07fb5f9f9b8d295ead2c4fd58d3f1f17451ca98dd73e0bbfeaed 2024-04-26T22:39:33.739 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T22:39:33.739 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T22:39:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T22:39:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T22:39:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T22:39:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T22:39:33.757 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T22:39:33.757 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T22:39:33.765 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T22:39:33.765 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T22:39:33.765 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T22:39:33.765 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T22:39:33.765 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr to start... 2024-04-26T22:39:33.765 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr... 2024-04-26T22:39:33.990 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:33 smithi001 systemd[1]: Started Ceph mgr.x for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:39:33.991 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:33 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:33.818+0000 7f6a124e7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T22:39:34.241 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:34.013+0000 7f6a124e7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:39:34.241 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:34.119+0000 7f6a124e7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T22:39:34.550 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1497790592' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T22:39:34.550 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:34.461+0000 7f6a124e7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T22:39:34.551 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:34.549+0000 7f6a124e7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T22:39:34.895 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:34.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:39:34.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:34.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T22:39:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T22:39:34.909 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T22:39:34.910 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:34.911 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T22:39:26.837351+0000", 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.231+0000 7f5255446700 1 Processor -- start 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.231+0000 7f5255446700 1 -- start start 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.232+0000 7f5255446700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f52501091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.232+0000 7f5255446700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f52501097c0 con 0x7f5250108dd0 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.232+0000 7f524effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f52501091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.232+0000 7f524effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f52501091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43660/0 (socket says 172.21.15.1:43660) 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.232+0000 7f524effd700 1 -- 172.21.15.1:0/2840191961 learned_addr learned my addr 172.21.15.1:0/2840191961 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.232+0000 7f524effd700 1 -- 172.21.15.1:0/2840191961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5250109f80 con 0x7f5250108dd0 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.233+0000 7f524effd700 1 --2- 172.21.15.1:0/2840191961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f52501091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f52400098a0 tx=0x7f5240009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbf2817aac9b55b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.233+0000 7f524dffb700 1 -- 172.21.15.1:0/2840191961 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5240004160 con 0x7f5250108dd0 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.233+0000 7f524dffb700 1 -- 172.21.15.1:0/2840191961 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5240004300 con 0x7f5250108dd0 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.233+0000 7f524dffb700 1 -- 172.21.15.1:0/2840191961 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5240011af0 con 0x7f5250108dd0 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.233+0000 7f5255446700 1 -- 172.21.15.1:0/2840191961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 msgr2=0x7f52501091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.233+0000 7f5255446700 1 --2- 172.21.15.1:0/2840191961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f52501091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f52400098a0 tx=0x7f5240009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 -- 172.21.15.1:0/2840191961 shutdown_connections 2024-04-26T22:39:34.913 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 --2- 172.21.15.1:0/2840191961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f52501091f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 -- 172.21.15.1:0/2840191961 >> 172.21.15.1:0/2840191961 conn(0x7f5250104370 msgr2=0x7f52501067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 -- 172.21.15.1:0/2840191961 shutdown_connections 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 -- 172.21.15.1:0/2840191961 wait complete. 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 Processor -- start 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.234+0000 7f5255446700 1 -- start start 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f5255446700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f525007f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f5255446700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f525010d330 con 0x7f5250108dd0 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f524effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f525007f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f524effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f525007f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43674/0 (socket says 172.21.15.1:43674) 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f524effd700 1 -- 172.21.15.1:0/1497790592 learned_addr learned my addr 172.21.15.1:0/1497790592 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f524effd700 1 -- 172.21.15.1:0/1497790592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525007f930 con 0x7f5250108dd0 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f524effd700 1 --2- 172.21.15.1:0/1497790592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f525007f3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5240009fb0 tx=0x7f5240011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:34.914 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52400036a0 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525007db20 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f524001a430 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.235+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5240019550 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.236+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525007e060 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.236+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5240019b90 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.236+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5240024850 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.237+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5250064a80 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.240+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f5240043ec0 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.325+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f525007e310 con 0x7f5250108dd0 2024-04-26T22:39:34.915 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.326+0000 7f5247fff700 1 -- 172.21.15.1:0/1497790592 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5240032020 con 0x7f5250108dd0 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.327+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 msgr2=0x7f525007f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.327+0000 7f5255446700 1 --2- 172.21.15.1:0/1497790592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f525007f3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5240009fb0 tx=0x7f5240011940 comp rx=0 tx=0).stop 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.327+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 shutdown_connections 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.327+0000 7f5255446700 1 --2- 172.21.15.1:0/1497790592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5250108dd0 0x7f525007f3f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.327+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 >> 172.21.15.1:0/1497790592 conn(0x7f5250104370 msgr2=0x7f5250104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.327+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 shutdown_connections 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:34.328+0000 7f5255446700 1 -- 172.21.15.1:0/1497790592 wait complete. 2024-04-26T22:39:34.916 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (1/15)... 2024-04-26T22:39:36.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:35.812+0000 7f6a124e7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T22:39:36.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:35.987+0000 7f6a124e7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T22:39:36.462 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:36.162+0000 7f6a124e7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T22:39:36.462 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:36.462+0000 7f6a124e7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T22:39:36.767 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:36.557+0000 7f6a124e7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T22:39:36.767 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:36.766+0000 7f6a124e7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T22:39:37.022 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:36.974+0000 7f6a124e7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:39:37.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:37 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/84244647' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T22:39:38.033 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:37 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:37.631+0000 7f6a124e7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T22:39:38.038 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T22:39:38.039 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:38.040 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T22:39:38.040 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T22:39:38.040 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T22:39:38.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T22:39:38.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T22:39:38.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T22:39:38.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:38.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T22:39:26.837351+0000", 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a3de8700 1 Processor -- start 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a3de8700 1 -- start start 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a3de8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c10bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a3de8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f469c07cea0 con 0x7f469c109800 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a1b84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c10bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a1b84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c10bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43684/0 (socket says 172.21.15.1:43684) 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.346+0000 7f46a1b84700 1 -- 172.21.15.1:0/2627218130 learned_addr learned my addr 172.21.15.1:0/2627218130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.347+0000 7f46a1b84700 1 -- 172.21.15.1:0/2627218130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c10c130 con 0x7f469c109800 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.347+0000 7f46a1b84700 1 --2- 172.21.15.1:0/2627218130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c10bbf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46980098a0 tx=0x7f4698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d96c0335226c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:38.056 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.347+0000 7f46a0b82700 1 -- 172.21.15.1:0/2627218130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4698004160 con 0x7f469c109800 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.347+0000 7f46a0b82700 1 -- 172.21.15.1:0/2627218130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4698004300 con 0x7f469c109800 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.347+0000 7f46a0b82700 1 -- 172.21.15.1:0/2627218130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4698011af0 con 0x7f469c109800 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 -- 172.21.15.1:0/2627218130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 msgr2=0x7f469c10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 --2- 172.21.15.1:0/2627218130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c10bbf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46980098a0 tx=0x7f4698009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 -- 172.21.15.1:0/2627218130 shutdown_connections 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 --2- 172.21.15.1:0/2627218130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c10bbf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 -- 172.21.15.1:0/2627218130 >> 172.21.15.1:0/2627218130 conn(0x7f469c1031f0 msgr2=0x7f469c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 -- 172.21.15.1:0/2627218130 shutdown_connections 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 -- 172.21.15.1:0/2627218130 wait complete. 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 Processor -- start 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.348+0000 7f46a3de8700 1 -- start start 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a3de8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a3de8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f469c10d100 con 0x7f469c109800 2024-04-26T22:39:38.057 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a1b84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a1b84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c19cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43688/0 (socket says 172.21.15.1:43688) 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a1b84700 1 -- 172.21.15.1:0/84244647 learned_addr learned my addr 172.21.15.1:0/84244647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a1b84700 1 -- 172.21.15.1:0/84244647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c19d220 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a1b84700 1 --2- 172.21.15.1:0/84244647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c19cce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4698011ac0 tx=0x7f4698011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4698003970 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4698003b10 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c19d4b0 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4698024610 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.349+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c1a0190 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.350+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4698024bd0 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.350+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4698019700 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.351+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469c064a80 con 0x7f469c109800 2024-04-26T22:39:38.058 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.353+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4698043ec0 con 0x7f469c109800 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.443+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f469c1a0360 con 0x7f469c109800 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.443+0000 7f468effd700 1 -- 172.21.15.1:0/84244647 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4698032030 con 0x7f469c109800 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 msgr2=0x7f469c19cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 --2- 172.21.15.1:0/84244647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c19cce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4698011ac0 tx=0x7f4698011ba0 comp rx=0 tx=0).stop 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 shutdown_connections 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 --2- 172.21.15.1:0/84244647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c109800 0x7f469c19cce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 >> 172.21.15.1:0/84244647 conn(0x7f469c1031f0 msgr2=0x7f469c103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 shutdown_connections 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:37.444+0000 7f46a3de8700 1 -- 172.21.15.1:0/84244647 wait complete. 2024-04-26T22:39:38.059 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (2/15)... 2024-04-26T22:39:38.346 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:38.073+0000 7f6a124e7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T22:39:38.346 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:38.173+0000 7f6a124e7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T22:39:38.846 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:38.383+0000 7f6a124e7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T22:39:39.273 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:38.926+0000 7f6a124e7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T22:39:39.273 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:39.021+0000 7f6a124e7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T22:39:39.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:39.272+0000 7f6a124e7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T22:39:39.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:39.410+0000 7f6a124e7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T22:39:39.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:39.508+0000 7f6a124e7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T22:39:39.939 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:39.628+0000 7f6a124e7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T22:39:40.268 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:39.938+0000 7f6a124e7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T22:39:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:40 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2714192919' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T22:39:40.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:40.586+0000 7f6a124e7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T22:39:40.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:40.817+0000 7f6a124e7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:40.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T22:39:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T22:39:40.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:40.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T22:39:40.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:40.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T22:39:40.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T22:39:41.008 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T22:39:41.009 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T22:39:26.837351+0000", 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:41.010 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.468+0000 7f1885b58700 1 Processor -- start 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.468+0000 7f1885b58700 1 -- start start 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f1885b58700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f18801080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f1885b58700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1880108680 con 0x7f1880107c90 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f187f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f18801080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f187f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f18801080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43692/0 (socket says 172.21.15.1:43692) 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f187f7fe700 1 -- 172.21.15.1:0/97827840 learned_addr learned my addr 172.21.15.1:0/97827840 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f187f7fe700 1 -- 172.21.15.1:0/97827840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1880108e90 con 0x7f1880107c90 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f187f7fe700 1 --2- 172.21.15.1:0/97827840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f18801080b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18700098a0 tx=0x7f1870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea6956ada9fc2ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.469+0000 7f187e7fc700 1 -- 172.21.15.1:0/97827840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1870004160 con 0x7f1880107c90 2024-04-26T22:39:41.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f187e7fc700 1 -- 172.21.15.1:0/97827840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1870004300 con 0x7f1880107c90 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f187e7fc700 1 -- 172.21.15.1:0/97827840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1870011af0 con 0x7f1880107c90 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 -- 172.21.15.1:0/97827840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 msgr2=0x7f18801080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 --2- 172.21.15.1:0/97827840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f18801080b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18700098a0 tx=0x7f1870009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 -- 172.21.15.1:0/97827840 shutdown_connections 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 --2- 172.21.15.1:0/97827840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f18801080b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 -- 172.21.15.1:0/97827840 >> 172.21.15.1:0/97827840 conn(0x7f18801031f0 msgr2=0x7f1880105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 -- 172.21.15.1:0/97827840 shutdown_connections 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.470+0000 7f1885b58700 1 -- 172.21.15.1:0/97827840 wait complete. 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f1885b58700 1 Processor -- start 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f1885b58700 1 -- start start 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f1885b58700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f188007ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f1885b58700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f188010c240 con 0x7f1880107c90 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f187f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f188007ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f187f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f188007ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43706/0 (socket says 172.21.15.1:43706) 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.471+0000 7f187f7fe700 1 -- 172.21.15.1:0/2714192919 learned_addr learned my addr 172.21.15.1:0/2714192919 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187f7fe700 1 -- 172.21.15.1:0/2714192919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188019fa70 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187f7fe700 1 --2- 172.21.15.1:0/2714192919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f188007ca60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1870009fb0 tx=0x7f1870011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18700036a0 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f187001a430 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188019fc40 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1870019550 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18801a0060 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1870019be0 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.472+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1870024730 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.473+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1880064a80 con 0x7f1880107c90 2024-04-26T22:39:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.475+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f1870043ec0 con 0x7f1880107c90 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.550+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f18801a0750 con 0x7f1880107c90 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.550+0000 7f187cff9700 1 -- 172.21.15.1:0/2714192919 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1870032030 con 0x7f1880107c90 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.551+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 msgr2=0x7f188007ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.551+0000 7f1885b58700 1 --2- 172.21.15.1:0/2714192919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f188007ca60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1870009fb0 tx=0x7f1870011940 comp rx=0 tx=0).stop 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.551+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 shutdown_connections 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.551+0000 7f1885b58700 1 --2- 172.21.15.1:0/2714192919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1880107c90 0x7f188007ca60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.551+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 >> 172.21.15.1:0/2714192919 conn(0x7f18801031f0 msgr2=0x7f1880103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.552+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 shutdown_connections 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:40.552+0000 7f1885b58700 1 -- 172.21.15.1:0/2714192919 wait complete. 2024-04-26T22:39:41.014 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (3/15)... 2024-04-26T22:39:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: Activating manager daemon x 2024-04-26T22:39:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: mgrmap e2: x(active, starting, since 0.00316119s) 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: Manager daemon x is now available 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' 2024-04-26T22:39:42.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:41 smithi001 ceph-mon[29435]: from='mgr.14100 172.21.15.1:0/1674768003' entity='mgr.x' 2024-04-26T22:39:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:42 smithi001 ceph-mon[29435]: mgrmap e3: x(active, since 1.00743s) 2024-04-26T22:39:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:43 smithi001 ceph-mon[29435]: mgrmap e4: x(active, since 2s) 2024-04-26T22:39:46.625 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2018711033' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T22:39:52.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:52.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:52.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T22:39:52.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T22:39:52.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T22:39:26.837351+0000", 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T22:39:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.146+0000 7f6553de6700 1 Processor -- start 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.146+0000 7f6553de6700 1 -- start start 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.146+0000 7f6553de6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.146+0000 7f6553de6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f654c1097c0 con 0x7f654c108dd0 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6551b82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6551b82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48030/0 (socket says 172.21.15.1:48030) 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6551b82700 1 -- 172.21.15.1:0/444231692 learned_addr learned my addr 172.21.15.1:0/444231692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:52.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6551b82700 1 -- 172.21.15.1:0/444231692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654c109f80 con 0x7f654c108dd0 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6551b82700 1 --2- 172.21.15.1:0/444231692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c1091f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f653c009e80 tx=0x7f653c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6b3760b4561ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6550b80700 1 -- 172.21.15.1:0/444231692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c017070 con 0x7f654c108dd0 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6550b80700 1 -- 172.21.15.1:0/444231692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f653c00b8c0 con 0x7f654c108dd0 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.147+0000 7f6550b80700 1 -- 172.21.15.1:0/444231692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c004490 con 0x7f654c108dd0 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 -- 172.21.15.1:0/444231692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 msgr2=0x7f654c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 --2- 172.21.15.1:0/444231692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c1091f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f653c009e80 tx=0x7f653c00b330 comp rx=0 tx=0).stop 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 -- 172.21.15.1:0/444231692 shutdown_connections 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 --2- 172.21.15.1:0/444231692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c1091f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 -- 172.21.15.1:0/444231692 >> 172.21.15.1:0/444231692 conn(0x7f654c104370 msgr2=0x7f654c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 -- 172.21.15.1:0/444231692 shutdown_connections 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.148+0000 7f6553de6700 1 -- 172.21.15.1:0/444231692 wait complete. 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6553de6700 1 Processor -- start 2024-04-26T22:39:52.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6553de6700 1 -- start start 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6553de6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6553de6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f654c10d330 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6551b82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6551b82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c19ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48040/0 (socket says 172.21.15.1:48040) 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6551b82700 1 -- 172.21.15.1:0/2018711033 learned_addr learned my addr 172.21.15.1:0/2018711033 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6551b82700 1 -- 172.21.15.1:0/2018711033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654c19d210 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.149+0000 7f6551b82700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c19ccd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f653c004a60 tx=0x7f653c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.150+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c020020 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.150+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f653c012430 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.150+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654c19d4a0 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.150+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c017040 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.150+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f654c1a0180 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.151+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f653c023020 con 0x7f654c108dd0 2024-04-26T22:39:52.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.150+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f654c064a80 con 0x7f654c108dd0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.151+0000 7f654affd700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f65340395b0 0x7f653403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.151+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f653c00f050 con 0x7f654c108dd0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.151+0000 7f6551381700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f65340395b0 0x7f653403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.151+0000 7f6551381700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f65340395b0 0x7f653403ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6540006fd0 tx=0x7f6540006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.156+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f653c02a020 con 0x7f654c108dd0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.449+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f654c1a0350 con 0x7f654c108dd0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.450+0000 7f654affd700 1 -- 172.21.15.1:0/2018711033 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f654c1a0350 con 0x7f654c108dd0 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f65340395b0 msgr2=0x7f653403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f65340395b0 0x7f653403ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6540006fd0 tx=0x7f6540006e40 comp rx=0 tx=0).stop 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 msgr2=0x7f654c19ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c19ccd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f653c004a60 tx=0x7f653c00bd50 comp rx=0 tx=0).stop 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 shutdown_connections 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f65340395b0 0x7f653403ba70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:52.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 --2- 172.21.15.1:0/2018711033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f654c108dd0 0x7f654c19ccd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:52.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 >> 172.21.15.1:0/2018711033 conn(0x7f654c104370 msgr2=0x7f654c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:52.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 shutdown_connections 2024-04-26T22:39:52.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:46.452+0000 7f6553de6700 1 -- 172.21.15.1:0/2018711033 wait complete. 2024-04-26T22:39:52.997 INFO:teuthology.orchestra.run.smithi001.stdout:mgr is available 2024-04-26T22:39:54.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:54 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4003521876' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T22:39:54.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:54.731 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T22:39:54.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.962+0000 7fc773914700 1 Processor -- start 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc773914700 1 -- start start 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc773914700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc773914700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c109740 con 0x7fc76c108d50 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc7716b0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc7716b0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37204/0 (socket says 172.21.15.1:37204) 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc7716b0700 1 -- 172.21.15.1:0/1400118160 learned_addr learned my addr 172.21.15.1:0/1400118160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.963+0000 7fc7716b0700 1 -- 172.21.15.1:0/1400118160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c109f50 con 0x7fc76c108d50 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc7716b0700 1 --2- 172.21.15.1:0/1400118160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c109170 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0098a0 tx=0x7fc75c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e919e3dfa122afd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc763fff700 1 -- 172.21.15.1:0/1400118160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc75c004160 con 0x7fc76c108d50 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc763fff700 1 -- 172.21.15.1:0/1400118160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc75c004300 con 0x7fc76c108d50 2024-04-26T22:39:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc763fff700 1 -- 172.21.15.1:0/1400118160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc75c011af0 con 0x7fc76c108d50 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc773914700 1 -- 172.21.15.1:0/1400118160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 msgr2=0x7fc76c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc773914700 1 --2- 172.21.15.1:0/1400118160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c109170 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0098a0 tx=0x7fc75c009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc773914700 1 -- 172.21.15.1:0/1400118160 shutdown_connections 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc773914700 1 --2- 172.21.15.1:0/1400118160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c109170 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc773914700 1 -- 172.21.15.1:0/1400118160 >> 172.21.15.1:0/1400118160 conn(0x7fc76c1042d0 msgr2=0x7fc76c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.964+0000 7fc773914700 1 -- 172.21.15.1:0/1400118160 shutdown_connections 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc773914700 1 -- 172.21.15.1:0/1400118160 wait complete. 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc773914700 1 Processor -- start 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc773914700 1 -- start start 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc773914700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c07f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc773914700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c10d300 con 0x7fc76c108d50 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc7716b0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c07f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc7716b0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c07f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37208/0 (socket says 172.21.15.1:37208) 2024-04-26T22:39:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.965+0000 7fc7716b0700 1 -- 172.21.15.1:0/4003521876 learned_addr learned my addr 172.21.15.1:0/4003521876 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc7716b0700 1 -- 172.21.15.1:0/4003521876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c07f930 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc7716b0700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c07f3f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0117b0 tx=0x7fc75c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc75c003af0 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc75c020070 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c07db20 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc75c019630 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.966+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c07e060 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.967+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fc75c01a430 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.967+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c064a80 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.967+0000 7fc7627fc700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fc758039560 0x7fc75803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.967+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc75c04db20 con 0x7fc76c108d50 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.967+0000 7fc770eaf700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fc758039560 0x7fc75803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:54.735 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.968+0000 7fc770eaf700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fc758039560 0x7fc75803ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc768006fd0 tx=0x7fc768006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:53.972+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc75c02b260 con 0x7fc76c108d50 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.178+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc76c07e850 con 0x7fc76c108d50 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.179+0000 7fc7627fc700 1 -- 172.21.15.1:0/4003521876 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fc75c04d540 con 0x7fc76c108d50 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fc758039560 msgr2=0x7fc75803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fc758039560 0x7fc75803ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc768006fd0 tx=0x7fc768006e40 comp rx=0 tx=0).stop 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 msgr2=0x7fc76c07f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c07f3f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0117b0 tx=0x7fc75c011b70 comp rx=0 tx=0).stop 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 shutdown_connections 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fc758039560 0x7fc75803ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 --2- 172.21.15.1:0/4003521876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc76c108d50 0x7fc76c07f3f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 >> 172.21.15.1:0/4003521876 conn(0x7fc76c1042d0 msgr2=0x7fc76c104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 shutdown_connections 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:54.181+0000 7fc773914700 1 -- 172.21.15.1:0/4003521876 wait complete. 2024-04-26T22:39:54.736 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling cephadm module... 2024-04-26T22:39:55.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:55 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1275715830' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T22:39:55.796 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: ignoring --setuser ceph since I am not root 2024-04-26T22:39:55.796 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: ignoring --setgroup ceph since I am not root 2024-04-26T22:39:55.796 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:55.666+0000 7fdfb5993200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T22:39:56.078 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:55.859+0000 7fdfb5993200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:39:56.079 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:55.965+0000 7fdfb5993200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T22:39:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.156+0000 7f4a7d2f1700 1 Processor -- start 2024-04-26T22:39:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a7d2f1700 1 -- start start 2024-04-26T22:39:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a7d2f1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a781091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a7d2f1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a781097c0 con 0x7f4a78108dd0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a76ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a781091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a76ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a781091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37216/0 (socket says 172.21.15.1:37216) 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a76ffd700 1 -- 172.21.15.1:0/2455155551 learned_addr learned my addr 172.21.15.1:0/2455155551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a76ffd700 1 -- 172.21.15.1:0/2455155551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a78109f80 con 0x7f4a78108dd0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.157+0000 7f4a76ffd700 1 --2- 172.21.15.1:0/2455155551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a781091f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098a0 tx=0x7f4a68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faac8e0e42964ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a75ffb700 1 -- 172.21.15.1:0/2455155551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a68004160 con 0x7f4a78108dd0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a75ffb700 1 -- 172.21.15.1:0/2455155551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a68004300 con 0x7f4a78108dd0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a75ffb700 1 -- 172.21.15.1:0/2455155551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a68011af0 con 0x7f4a78108dd0 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/2455155551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 msgr2=0x7f4a781091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 --2- 172.21.15.1:0/2455155551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a781091f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098a0 tx=0x7f4a68009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/2455155551 shutdown_connections 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 --2- 172.21.15.1:0/2455155551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a781091f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/2455155551 >> 172.21.15.1:0/2455155551 conn(0x7f4a78104370 msgr2=0x7f4a781067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/2455155551 shutdown_connections 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.158+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/2455155551 wait complete. 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a7d2f1700 1 Processor -- start 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a7d2f1700 1 -- start start 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a7d2f1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a7819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a7d2f1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7810d330 con 0x7f4a78108dd0 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a76ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a7819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a76ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a7819cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37224/0 (socket says 172.21.15.1:37224) 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.159+0000 7f4a76ffd700 1 -- 172.21.15.1:0/1275715830 learned_addr learned my addr 172.21.15.1:0/1275715830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a76ffd700 1 -- 172.21.15.1:0/1275715830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7819d250 con 0x7f4a78108dd0 2024-04-26T22:39:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a76ffd700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a7819cd10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a680117b0 tx=0x7f4a68011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a68003af0 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a68020070 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7819d4e0 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a68019630 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.160+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a781a01c0 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.161+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4a680198a0 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.161+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a78064a80 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.161+0000 7f4a5ffff700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f4a600395b0 0x7f4a6003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.161+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4a6804dbd0 con 0x7f4a78108dd0 2024-04-26T22:39:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.161+0000 7f4a767fc700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f4a600395b0 0x7f4a6003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.161+0000 7f4a767fc700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f4a600395b0 0x7f4a6003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c006fd0 tx=0x7f4a6c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.166+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a680042a0 con 0x7f4a78108dd0 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.434+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f4a781a0390 con 0x7f4a78108dd0 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.485+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f4a781a0390 con 0x7f4a78108dd0 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.487+0000 7f4a5ffff700 1 -- 172.21.15.1:0/1275715830 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f4a68019b90 con 0x7f4a78108dd0 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f4a600395b0 msgr2=0x7f4a6003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f4a600395b0 0x7f4a6003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c006fd0 tx=0x7f4a6c006e40 comp rx=0 tx=0).stop 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 msgr2=0x7f4a7819cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a7819cd10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a680117b0 tx=0x7f4a68011b70 comp rx=0 tx=0).stop 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 shutdown_connections 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f4a600395b0 0x7f4a6003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 --2- 172.21.15.1:0/1275715830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78108dd0 0x7f4a7819cd10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.488+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 >> 172.21.15.1:0/1275715830 conn(0x7f4a78104370 msgr2=0x7f4a78104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:56.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.489+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 shutdown_connections 2024-04-26T22:39:56.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:55.489+0000 7f4a7d2f1700 1 -- 172.21.15.1:0/1275715830 wait complete. 2024-04-26T22:39:56.346 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:56 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:56.290+0000 7fdfb5993200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T22:39:56.748 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:56 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:56.379+0000 7fdfb5993200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T22:39:56.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:56 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1275715830' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T22:39:56.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:56 smithi001 ceph-mon[29435]: mgrmap e5: x(active, since 14s) 2024-04-26T22:39:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:39:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T22:39:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T22:39:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T22:39:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T22:39:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.541+0000 7f36dbe55700 1 Processor -- start 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36dbe55700 1 -- start start 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36dbe55700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d410bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36dbe55700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f36d407cdc0 con 0x7f36d41098e0 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36d9bf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d410bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36d9bf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d410bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37250/0 (socket says 172.21.15.1:37250) 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36d9bf1700 1 -- 172.21.15.1:0/3827157386 learned_addr learned my addr 172.21.15.1:0/3827157386 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.542+0000 7f36d9bf1700 1 -- 172.21.15.1:0/3827157386 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d410c210 con 0x7f36d41098e0 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.543+0000 7f36d9bf1700 1 --2- 172.21.15.1:0/3827157386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d410bcd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f36d00098a0 tx=0x7f36d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3192425a30121c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.543+0000 7f36d8bef700 1 -- 172.21.15.1:0/3827157386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d0004160 con 0x7f36d41098e0 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.543+0000 7f36d8bef700 1 -- 172.21.15.1:0/3827157386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f36d0004300 con 0x7f36d41098e0 2024-04-26T22:39:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.543+0000 7f36d8bef700 1 -- 172.21.15.1:0/3827157386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d0011af0 con 0x7f36d41098e0 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 -- 172.21.15.1:0/3827157386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 msgr2=0x7f36d410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 --2- 172.21.15.1:0/3827157386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d410bcd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f36d00098a0 tx=0x7f36d0009bb0 comp rx=0 tx=0).stop 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 -- 172.21.15.1:0/3827157386 shutdown_connections 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 --2- 172.21.15.1:0/3827157386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d410bcd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 -- 172.21.15.1:0/3827157386 >> 172.21.15.1:0/3827157386 conn(0x7f36d41031f0 msgr2=0x7f36d4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 -- 172.21.15.1:0/3827157386 shutdown_connections 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.544+0000 7f36dbe55700 1 -- 172.21.15.1:0/3827157386 wait complete. 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36dbe55700 1 Processor -- start 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36dbe55700 1 -- start start 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36dbe55700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36dbe55700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f36d410d110 con 0x7f36d41098e0 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36d9bf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36d9bf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d419ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37252/0 (socket says 172.21.15.1:37252) 2024-04-26T22:39:57.478 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36d9bf1700 1 -- 172.21.15.1:0/3909191933 learned_addr learned my addr 172.21.15.1:0/3909191933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.545+0000 7f36d9bf1700 1 -- 172.21.15.1:0/3909191933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d419d230 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.546+0000 7f36d9bf1700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d419ccf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f36d0011ac0 tx=0x7f36d0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.546+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d0003970 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.546+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f36d0003b10 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.546+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d419d4c0 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.546+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d0024610 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.546+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d41a01a0 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.547+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f36d0024840 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.547+0000 7f36caffd700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 0x7f36c003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.547+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f36d004d8e0 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.547+0000 7f36d93f0700 1 -- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 msgr2=0x7f36c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.547+0000 7f36d93f0700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 0x7f36c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.547+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d4064a80 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.552+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f36d00172b0 con 0x7f36d41098e0 2024-04-26T22:39:57.479 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.748+0000 7f36d93f0700 1 -- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 msgr2=0x7f36c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.748+0000 7f36d93f0700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 0x7f36c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.881+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f36d41a0370 con 0x7f36d41098e0 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.881+0000 7f36caffd700 1 -- 172.21.15.1:0/3909191933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f36d002d910 con 0x7f36d41098e0 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 msgr2=0x7f36c003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 0x7f36c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 msgr2=0x7f36d419ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d419ccf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f36d0011ac0 tx=0x7f36d0011ba0 comp rx=0 tx=0).stop 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 shutdown_connections 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7f36c0039600 0x7f36c003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 --2- 172.21.15.1:0/3909191933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d41098e0 0x7f36d419ccf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 >> 172.21.15.1:0/3909191933 conn(0x7f36d41031f0 msgr2=0x7f36d4103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 shutdown_connections 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:56.884+0000 7f36dbe55700 1 -- 172.21.15.1:0/3909191933 wait complete. 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-04-26T22:39:57.480 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 5... 2024-04-26T22:39:57.787 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:39:57 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3909191933' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T22:39:57.787 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:57.650+0000 7fdfb5993200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T22:39:58.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:57.835+0000 7fdfb5993200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T22:39:58.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:58.013+0000 7fdfb5993200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T22:39:58.409 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:58.312+0000 7fdfb5993200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T22:39:58.409 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:58.408+0000 7fdfb5993200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T22:39:58.819 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:58.618+0000 7fdfb5993200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T22:39:59.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:58.818+0000 7fdfb5993200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:39:59.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:59.445+0000 7fdfb5993200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T22:40:00.193 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:59.882+0000 7fdfb5993200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T22:40:00.194 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:39:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:39:59.983+0000 7fdfb5993200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T22:40:00.194 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:00.193+0000 7fdfb5993200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T22:40:01.085 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:00.737+0000 7fdfb5993200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T22:40:01.085 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:00.833+0000 7fdfb5993200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T22:40:01.347 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:01.084+0000 7fdfb5993200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T22:40:01.347 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:01.222+0000 7fdfb5993200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T22:40:01.348 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:01.321+0000 7fdfb5993200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T22:40:01.751 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:01.441+0000 7fdfb5993200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T22:40:01.752 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:01.750+0000 7fdfb5993200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T22:40:02.659 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:02.382+0000 7fdfb5993200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T22:40:02.659 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:02.605+0000 7fdfb5993200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T22:40:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: Active manager daemon x restarted 2024-04-26T22:40:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: Activating manager daemon x 2024-04-26T22:40:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: mgrmap e6: x(active, starting, since 0.00355392s) 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: Manager daemon x is now available 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:02 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T22:40:03.974 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:03 smithi001 ceph-mon[29435]: Found migration_current of "None". Setting to last migration. 2024-04-26T22:40:03.974 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:03 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T22:40:03.974 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:03 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:03.974 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:03 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:03.975 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:03 smithi001 ceph-mon[29435]: mgrmap e7: x(active, since 1.00545s) 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.891+0000 7fbfc7a01700 1 Processor -- start 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.891+0000 7fbfc7a01700 1 -- start start 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.891+0000 7fbfc7a01700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc010bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:04.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.891+0000 7fbfc7a01700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc007cea0 con 0x7fbfc0109800 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfc579d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc010bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfc579d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc010bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37262/0 (socket says 172.21.15.1:37262) 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfc579d700 1 -- 172.21.15.1:0/3526816352 learned_addr learned my addr 172.21.15.1:0/3526816352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfc579d700 1 -- 172.21.15.1:0/3526816352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc010c130 con 0x7fbfc0109800 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfc579d700 1 --2- 172.21.15.1:0/3526816352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc010bbf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00098a0 tx=0x7fbfb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1480338187a5fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfb7fff700 1 -- 172.21.15.1:0/3526816352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfb0004160 con 0x7fbfc0109800 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfb7fff700 1 -- 172.21.15.1:0/3526816352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbfb0004300 con 0x7fbfc0109800 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.892+0000 7fbfb7fff700 1 -- 172.21.15.1:0/3526816352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfb0011af0 con 0x7fbfc0109800 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 -- 172.21.15.1:0/3526816352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 msgr2=0x7fbfc010bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/3526816352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc010bbf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00098a0 tx=0x7fbfb0009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 -- 172.21.15.1:0/3526816352 shutdown_connections 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/3526816352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc010bbf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 -- 172.21.15.1:0/3526816352 >> 172.21.15.1:0/3526816352 conn(0x7fbfc01031f0 msgr2=0x7fbfc0105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 -- 172.21.15.1:0/3526816352 shutdown_connections 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.893+0000 7fbfc7a01700 1 -- 172.21.15.1:0/3526816352 wait complete. 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc7a01700 1 Processor -- start 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc7a01700 1 -- start start 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc7a01700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc7a01700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc010d100 con 0x7fbfc0109800 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc579d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc579d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37266/0 (socket says 172.21.15.1:37266) 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.894+0000 7fbfc579d700 1 -- 172.21.15.1:0/1377808090 learned_addr learned my addr 172.21.15.1:0/1377808090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfc579d700 1 -- 172.21.15.1:0/1377808090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc019d270 con 0x7fbfc0109800 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfc579d700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc019cd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0011ac0 tx=0x7fbfb0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfb0003970 con 0x7fbfc0109800 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc019d500 con 0x7fbfc0109800 2024-04-26T22:40:04.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc01a01f0 con 0x7fbfc0109800 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbfb0003b10 con 0x7fbfc0109800 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.895+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfb0024610 con 0x7fbfc0109800 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.896+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fbfb00248b0 con 0x7fbfc0109800 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.896+0000 7fbfb67fc700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.896+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 --> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbfac03c180 con 0x7fbfac0395b0 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.896+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbfb004d8d0 con 0x7fbfc0109800 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.896+0000 7fbfc4f9c700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 msgr2=0x7fbfac03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:57.896+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:58.097+0000 7fbfc4f9c700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 msgr2=0x7fbfac03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:58.097+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:58.498+0000 7fbfc4f9c700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 msgr2=0x7fbfac03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:40:04.247 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:58.498+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:59.299+0000 7fbfc4f9c700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 msgr2=0x7fbfac03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:39:59.299+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:00.900+0000 7fbfc4f9c700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 msgr2=0x7fbfac03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/722669381 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:00.900+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:02.609+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fbfb0024ba0 con 0x7fbfc0109800 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:02.609+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 msgr2=0x7fbfac03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:02.609+0000 7fbfb67fc700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.610+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fbfb004f920 con 0x7fbfc0109800 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.610+0000 7fbfb67fc700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fbfac03cea0 0x7fbfac03f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.610+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbfac03c180 con 0x7fbfac03cea0 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.610+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fbfac03cea0 0x7fbfac03f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:04.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.611+0000 7fbfc4f9c700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fbfac03cea0 0x7fbfac03f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc003a10 tx=0x7fbfbc009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.612+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fbfac03c180 con 0x7fbfac03cea0 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.620+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbfa4002630 con 0x7fbfac03cea0 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.620+0000 7fbfb67fc700 1 -- 172.21.15.1:0/1377808090 <== mgr.14120 v2:172.21.15.1:6800/3267707627 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fbfa4002630 con 0x7fbfac03cea0 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fbfac03cea0 msgr2=0x7fbfac03f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fbfac03cea0 0x7fbfac03f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc003a10 tx=0x7fbfbc009290 comp rx=0 tx=0).stop 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 msgr2=0x7fbfc019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc019cd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0011ac0 tx=0x7fbfb0011ba0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 shutdown_connections 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fbfac03cea0 0x7fbfac03f290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0109800 0x7fbfc019cd30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 --2- 172.21.15.1:0/1377808090 >> [v2:172.21.15.1:6800/722669381,v1:172.21.15.1:6801/722669381] conn(0x7fbfac0395b0 0x7fbfac03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 >> 172.21.15.1:0/1377808090 conn(0x7fbfc01031f0 msgr2=0x7fbfc0103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:04.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 shutdown_connections 2024-04-26T22:40:04.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:03.621+0000 7fbfc7a01700 1 -- 172.21.15.1:0/1377808090 wait complete. 2024-04-26T22:40:04.250 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 5 is available 2024-04-26T22:40:04.250 INFO:teuthology.orchestra.run.smithi001.stdout:Setting orchestrator backend to cephadm... 2024-04-26T22:40:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T22:40:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T22:40:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:03] ENGINE Bus STARTING 2024-04-26T22:40:04.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:03] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T22:40:04.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:04] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T22:40:04.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:04] ENGINE Bus STARTED 2024-04-26T22:40:04.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:04 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.664+0000 7f3b4b2b1700 1 Processor -- start 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4b2b1700 1 -- start start 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4b2b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b44108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4b2b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44108660 con 0x7f3b44107c70 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4904d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b44108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4904d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b44108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48560/0 (socket says 172.21.15.1:48560) 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4904d700 1 -- 172.21.15.1:0/2740026339 learned_addr learned my addr 172.21.15.1:0/2740026339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4904d700 1 -- 172.21.15.1:0/2740026339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b44108e70 con 0x7f3b44107c70 2024-04-26T22:40:05.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.665+0000 7f3b4904d700 1 --2- 172.21.15.1:0/2740026339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b44108090 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b400098a0 tx=0x7f3b40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ea60d4fb4e5a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b37fff700 1 -- 172.21.15.1:0/2740026339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b40004160 con 0x7f3b44107c70 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b37fff700 1 -- 172.21.15.1:0/2740026339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3b40004300 con 0x7f3b44107c70 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b37fff700 1 -- 172.21.15.1:0/2740026339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b40011af0 con 0x7f3b44107c70 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/2740026339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 msgr2=0x7f3b44108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 --2- 172.21.15.1:0/2740026339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b44108090 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b400098a0 tx=0x7f3b40009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/2740026339 shutdown_connections 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 --2- 172.21.15.1:0/2740026339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b44108090 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/2740026339 >> 172.21.15.1:0/2740026339 conn(0x7f3b441031d0 msgr2=0x7f3b44105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/2740026339 shutdown_connections 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.666+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/2740026339 wait complete. 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4b2b1700 1 Processor -- start 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4b2b1700 1 -- start start 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4b2b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b4419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4b2b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4410c220 con 0x7f3b44107c70 2024-04-26T22:40:05.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4904d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b4419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4904d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b4419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48566/0 (socket says 172.21.15.1:48566) 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.667+0000 7f3b4904d700 1 -- 172.21.15.1:0/792190627 learned_addr learned my addr 172.21.15.1:0/792190627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b4904d700 1 -- 172.21.15.1:0/792190627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4419d280 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b4904d700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b4419cd40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b40009fb0 tx=0x7f3b40011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b400036a0 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3b4001a430 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4419d510 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b40019550 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.668+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b441a0200 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.669+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3b40019850 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.669+0000 7f3b367fc700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f3b30039560 0x7f3b3003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.669+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b44064a80 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.669+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3b4004d930 con 0x7f3b44107c70 2024-04-26T22:40:05.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.669+0000 7f3b4884c700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f3b30039560 0x7f3b3003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.669+0000 7f3b4884c700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f3b30039560 0x7f3b3003ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006fd0 tx=0x7f3b38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.675+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3b4000fda0 con 0x7f3b44107c70 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.900+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f3b4402e790 con 0x7f3b30039560 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.905+0000 7f3b367fc700 1 -- 172.21.15.1:0/792190627 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3b4402e790 con 0x7f3b30039560 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f3b30039560 msgr2=0x7f3b3003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f3b30039560 0x7f3b3003ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006fd0 tx=0x7f3b38006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 msgr2=0x7f3b4419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b4419cd40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b40009fb0 tx=0x7f3b40011940 comp rx=0 tx=0).stop 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 shutdown_connections 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f3b30039560 0x7f3b3003ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 --2- 172.21.15.1:0/792190627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b44107c70 0x7f3b4419cd40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 >> 172.21.15.1:0/792190627 conn(0x7f3b441031d0 msgr2=0x7f3b441040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 shutdown_connections 2024-04-26T22:40:05.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:04.907+0000 7f3b4b2b1700 1 -- 172.21.15.1:0/792190627 wait complete. 2024-04-26T22:40:05.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:05 smithi001 ceph-mon[29435]: mgrmap e8: x(active, since 2s) 2024-04-26T22:40:05.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:05 smithi001 ceph-mon[29435]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:05.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:05 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:05.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:05 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:06.762 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:06 smithi001 ceph-mon[29435]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:06.770 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.939+0000 7f780669a700 1 Processor -- start 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.939+0000 7f780669a700 1 -- start start 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.939+0000 7f780669a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f78001091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.939+0000 7f780669a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78001097c0 con 0x7f7800108dd0 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77fffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f78001091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77fffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f78001091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48568/0 (socket says 172.21.15.1:48568) 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77fffff700 1 -- 172.21.15.1:0/3436408381 learned_addr learned my addr 172.21.15.1:0/3436408381 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77fffff700 1 -- 172.21.15.1:0/3436408381 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7800109f80 con 0x7f7800108dd0 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77fffff700 1 --2- 172.21.15.1:0/3436408381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f78001091f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77f00098a0 tx=0x7f77f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe2cda166cbd6ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77feffd700 1 -- 172.21.15.1:0/3436408381 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77f0004160 con 0x7f7800108dd0 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77feffd700 1 -- 172.21.15.1:0/3436408381 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f77f0004300 con 0x7f7800108dd0 2024-04-26T22:40:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.940+0000 7f77feffd700 1 -- 172.21.15.1:0/3436408381 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77f0011af0 con 0x7f7800108dd0 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 -- 172.21.15.1:0/3436408381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 msgr2=0x7f78001091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 --2- 172.21.15.1:0/3436408381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f78001091f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77f00098a0 tx=0x7f77f0009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 -- 172.21.15.1:0/3436408381 shutdown_connections 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 --2- 172.21.15.1:0/3436408381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f78001091f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 -- 172.21.15.1:0/3436408381 >> 172.21.15.1:0/3436408381 conn(0x7f7800104370 msgr2=0x7f78001067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 -- 172.21.15.1:0/3436408381 shutdown_connections 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.941+0000 7f780669a700 1 -- 172.21.15.1:0/3436408381 wait complete. 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f780669a700 1 Processor -- start 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f780669a700 1 -- start start 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f780669a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f780019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f780669a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f780010d330 con 0x7f7800108dd0 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f77fffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f780019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f77fffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f780019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48574/0 (socket says 172.21.15.1:48574) 2024-04-26T22:40:06.772 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.942+0000 7f77fffff700 1 -- 172.21.15.1:0/2506313893 learned_addr learned my addr 172.21.15.1:0/2506313893 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f77fffff700 1 -- 172.21.15.1:0/2506313893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780019d270 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f77fffff700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f780019cd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f77f00117b0 tx=0x7f77f0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77f0003af0 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f77f0020070 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780019d500 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77f0019630 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.943+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78001a01f0 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.944+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f77f00198a0 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.944+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780006a540 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.944+0000 7f77fd7fa700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f77e8039600 0x7f77e803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.944+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f77f004dc10 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.944+0000 7f77ff7fe700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f77e8039600 0x7f77e803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.945+0000 7f77ff7fe700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f77e8039600 0x7f77e803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77f4006fd0 tx=0x7f77f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:05.949+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f77f000b480 con 0x7f7800108dd0 2024-04-26T22:40:06.773 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.160+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f78001055c0 con 0x7f77e8039600 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.161+0000 7f77fd7fa700 1 -- 172.21.15.1:0/2506313893 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f78001055c0 con 0x7f77e8039600 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f77e8039600 msgr2=0x7f77e803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f77e8039600 0x7f77e803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77f4006fd0 tx=0x7f77f4006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 msgr2=0x7f780019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f780019cd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f77f00117b0 tx=0x7f77f0011b70 comp rx=0 tx=0).stop 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 shutdown_connections 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f77e8039600 0x7f77e803bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 --2- 172.21.15.1:0/2506313893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7800108dd0 0x7f780019cd30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 >> 172.21.15.1:0/2506313893 conn(0x7f7800104370 msgr2=0x7f7800104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 shutdown_connections 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:06.163+0000 7f780669a700 1 -- 172.21.15.1:0/2506313893 wait complete. 2024-04-26T22:40:06.774 INFO:teuthology.orchestra.run.smithi001.stdout:Generating ssh key... 2024-04-26T22:40:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-mon[29435]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-mon[29435]: Generating ssh key... 2024-04-26T22:40:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: Generating public/private rsa key pair. 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: Your identification has been saved in /tmp/tmp9tk42dcj/key. 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: Your public key has been saved in /tmp/tmp9tk42dcj/key.pub. 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: The key fingerprint is: 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: SHA256:Q9oIIn77C172qERZOBd9af+BOz6bdCNtC/G/o/ejSL4 ceph-bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: The key's randomart image is: 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: +---[RSA 3072]----+ 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | .. . | 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | . .. + | 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: |. .o.o + . . | 2024-04-26T22:40:07.847 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: |.. .=. = o . | 2024-04-26T22:40:07.848 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | . + o S .o . | 2024-04-26T22:40:07.848 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | o . . o+. | 2024-04-26T22:40:07.848 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | + o .=.* | 2024-04-26T22:40:07.848 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | o = o +oB ++ | 2024-04-26T22:40:07.848 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: | o.+.. E+++o*| 2024-04-26T22:40:07.848 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: +----[SHA256]-----+ 2024-04-26T22:40:08.092 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.210+0000 7fc923b6b700 1 Processor -- start 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.210+0000 7fc923b6b700 1 -- start start 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc923b6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc923b6b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c109760 con 0x7fc91c108d70 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc921907700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc921907700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48580/0 (socket says 172.21.15.1:48580) 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc921907700 1 -- 172.21.15.1:0/1393798411 learned_addr learned my addr 172.21.15.1:0/1393798411 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc921907700 1 -- 172.21.15.1:0/1393798411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91c109f70 con 0x7fc91c108d70 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc921907700 1 --2- 172.21.15.1:0/1393798411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c109190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc90c009e80 tx=0x7fc90c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19315b8c0a1d75f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc920905700 1 -- 172.21.15.1:0/1393798411 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc90c017070 con 0x7fc91c108d70 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc920905700 1 -- 172.21.15.1:0/1393798411 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc90c00b8c0 con 0x7fc91c108d70 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.211+0000 7fc920905700 1 -- 172.21.15.1:0/1393798411 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc90c004490 con 0x7fc91c108d70 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 -- 172.21.15.1:0/1393798411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 msgr2=0x7fc91c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 --2- 172.21.15.1:0/1393798411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c109190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc90c009e80 tx=0x7fc90c00b330 comp rx=0 tx=0).stop 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 -- 172.21.15.1:0/1393798411 shutdown_connections 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 --2- 172.21.15.1:0/1393798411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c109190 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 -- 172.21.15.1:0/1393798411 >> 172.21.15.1:0/1393798411 conn(0x7fc91c104310 msgr2=0x7fc91c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 -- 172.21.15.1:0/1393798411 shutdown_connections 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.212+0000 7fc923b6b700 1 -- 172.21.15.1:0/1393798411 wait complete. 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc923b6b700 1 Processor -- start 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc923b6b700 1 -- start start 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc923b6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c07f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc923b6b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c10d320 con 0x7fc91c108d70 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc921907700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c07f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc921907700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c07f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48590/0 (socket says 172.21.15.1:48590) 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc921907700 1 -- 172.21.15.1:0/3176557574 learned_addr learned my addr 172.21.15.1:0/3176557574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:08.094 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.213+0000 7fc921907700 1 -- 172.21.15.1:0/3176557574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91c07f930 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.214+0000 7fc921907700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c07f3f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00b7b0 tx=0x7fc90c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.214+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc90c020020 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.214+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc90c012430 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.214+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc90c017040 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.214+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc91c07db20 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.214+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc91c07e060 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.215+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc90c023020 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.215+0000 7fc91affd700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fc9040395b0 0x7fc90403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.215+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc91c064a80 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.215+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc90c00f050 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.215+0000 7fc921106700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fc9040395b0 0x7fc90403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.215+0000 7fc921106700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fc9040395b0 0x7fc90403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc910006fd0 tx=0x7fc910006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.220+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc90c02a020 con 0x7fc91c108d70 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.431+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fc91c07e890 con 0x7fc9040395b0 2024-04-26T22:40:08.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.587+0000 7fc91affd700 1 -- 172.21.15.1:0/3176557574 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc91c07e890 con 0x7fc9040395b0 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fc9040395b0 msgr2=0x7fc90403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fc9040395b0 0x7fc90403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc910006fd0 tx=0x7fc910006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 msgr2=0x7fc91c07f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c07f3f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00b7b0 tx=0x7fc90c00be20 comp rx=0 tx=0).stop 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 shutdown_connections 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fc9040395b0 0x7fc90403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 --2- 172.21.15.1:0/3176557574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc91c108d70 0x7fc91c07f3f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 >> 172.21.15.1:0/3176557574 conn(0x7fc91c104310 msgr2=0x7fc91c104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 shutdown_connections 2024-04-26T22:40:08.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:07.590+0000 7fc923b6b700 1 -- 172.21.15.1:0/3176557574 wait complete. 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.522+0000 7f1a14e0a700 1 Processor -- start 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.522+0000 7f1a14e0a700 1 -- start start 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a14e0a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1007db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a14e0a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1007e130 con 0x7f1a1007f700 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1007db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1007db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48592/0 (socket says 172.21.15.1:48592) 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0e59c700 1 -- 172.21.15.1:0/3053280095 learned_addr learned my addr 172.21.15.1:0/3053280095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:09.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0e59c700 1 -- 172.21.15.1:0/3053280095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1007e2b0 con 0x7f1a1007f700 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0e59c700 1 --2- 172.21.15.1:0/3053280095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1007db60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009e80 tx=0x7f1a0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1981e81bd953c7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0d59a700 1 -- 172.21.15.1:0/3053280095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a00017070 con 0x7f1a1007f700 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0d59a700 1 -- 172.21.15.1:0/3053280095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1a0000b8c0 con 0x7f1a1007f700 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.523+0000 7f1a0d59a700 1 -- 172.21.15.1:0/3053280095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a00004490 con 0x7f1a1007f700 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3053280095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 msgr2=0x7f1a1007db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 --2- 172.21.15.1:0/3053280095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1007db60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009e80 tx=0x7f1a0000b330 comp rx=0 tx=0).stop 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3053280095 shutdown_connections 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 --2- 172.21.15.1:0/3053280095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1007db60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3053280095 >> 172.21.15.1:0/3053280095 conn(0x7f1a10104370 msgr2=0x7f1a10106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3053280095 shutdown_connections 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.524+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3053280095 wait complete. 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a14e0a700 1 Processor -- start 2024-04-26T22:40:09.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a14e0a700 1 -- start start 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a14e0a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a14e0a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1010f340 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a0e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a0e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1019ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48606/0 (socket says 172.21.15.1:48606) 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.525+0000 7f1a0e59c700 1 -- 172.21.15.1:0/3007603963 learned_addr learned my addr 172.21.15.1:0/3007603963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a0e59c700 1 -- 172.21.15.1:0/3007603963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1019f390 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a0e59c700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1019ee50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000b7b0 tx=0x7f1a0000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a00020020 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1019f620 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1a00003d10 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a00017040 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.526+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a101a2310 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.527+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1a00023020 con 0x7f1a1007f700 2024-04-26T22:40:09.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.527+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a10064a80 con 0x7f1a1007f700 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.527+0000 7f1a077fe700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f19fc0395b0 0x7f19fc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.527+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1a0000f050 con 0x7f1a1007f700 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.527+0000 7f1a0dd9b700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f19fc0395b0 0x7f19fc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.528+0000 7f1a0dd9b700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f19fc0395b0 0x7f19fc03ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.532+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1a0002a020 con 0x7f1a1007f700 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.744+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f1a101055f0 con 0x7f19fc0395b0 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.745+0000 7f1a077fe700 1 -- 172.21.15.1:0/3007603963 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f1a101055f0 con 0x7f19fc0395b0 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f19fc0395b0 msgr2=0x7f19fc03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f19fc0395b0 0x7f19fc03ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f8006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 msgr2=0x7f1a1019ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1019ee50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000b7b0 tx=0x7f1a0000be90 comp rx=0 tx=0).stop 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 shutdown_connections 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f19fc0395b0 0x7f19fc03ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 --2- 172.21.15.1:0/3007603963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1007f700 0x7f1a1019ee50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:09.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 >> 172.21.15.1:0/3007603963 conn(0x7f1a10104370 msgr2=0x7f1a10104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:09.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 shutdown_connections 2024-04-26T22:40:09.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:08.747+0000 7f1a14e0a700 1 -- 172.21.15.1:0/3007603963 wait complete. 2024-04-26T22:40:09.337 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T22:40:09.337 INFO:teuthology.orchestra.run.smithi001.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T22:40:09.337 INFO:teuthology.orchestra.run.smithi001.stdout:Adding host smithi001... 2024-04-26T22:40:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:09 smithi001 ceph-mon[29435]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:10.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:10 smithi001 ceph-mon[29435]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:11 smithi001 ceph-mon[29435]: Deploying cephadm binary to smithi001 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Added host 'smithi001' with addr '172.21.15.1' 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.759+0000 7f2250bce700 1 Processor -- start 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.759+0000 7f2250bce700 1 -- start start 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.759+0000 7f2250bce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c07db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.759+0000 7f2250bce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f224c07e0c0 con 0x7f224c07f680 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.759+0000 7f224a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:14.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c07db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48614/0 (socket says 172.21.15.1:48614) 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224a59c700 1 -- 172.21.15.1:0/2319097343 learned_addr learned my addr 172.21.15.1:0/2319097343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224a59c700 1 -- 172.21.15.1:0/2319097343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224c07e240 con 0x7f224c07f680 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224a59c700 1 --2- 172.21.15.1:0/2319097343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c07db80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f223c0098a0 tx=0x7f223c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7cb12494b2eca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224959a700 1 -- 172.21.15.1:0/2319097343 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f223c004160 con 0x7f224c07f680 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224959a700 1 -- 172.21.15.1:0/2319097343 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f223c004300 con 0x7f224c07f680 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.760+0000 7f224959a700 1 -- 172.21.15.1:0/2319097343 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f223c011af0 con 0x7f224c07f680 2024-04-26T22:40:14.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 -- 172.21.15.1:0/2319097343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 msgr2=0x7f224c07db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 --2- 172.21.15.1:0/2319097343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c07db80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f223c0098a0 tx=0x7f223c009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 -- 172.21.15.1:0/2319097343 shutdown_connections 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 --2- 172.21.15.1:0/2319097343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c07db80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 -- 172.21.15.1:0/2319097343 >> 172.21.15.1:0/2319097343 conn(0x7f224c104370 msgr2=0x7f224c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 -- 172.21.15.1:0/2319097343 shutdown_connections 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.761+0000 7f2250bce700 1 -- 172.21.15.1:0/2319097343 wait complete. 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f2250bce700 1 Processor -- start 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f2250bce700 1 -- start start 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f2250bce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f2250bce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f224c10f320 con 0x7f224c07f680 2024-04-26T22:40:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f224a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f224a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48626/0 (socket says 172.21.15.1:48626) 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f224a59c700 1 -- 172.21.15.1:0/1176713332 learned_addr learned my addr 172.21.15.1:0/1176713332 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.762+0000 7f224a59c700 1 -- 172.21.15.1:0/1176713332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224c19f400 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.763+0000 7f224a59c700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c19eec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f223c011ac0 tx=0x7f223c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.763+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f223c003970 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.763+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f223c003b10 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.763+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f223c024610 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.763+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f224c19f690 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.763+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f224c1a2380 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.764+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f223c024890 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.764+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f224c064a80 con 0x7f224c07f680 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.764+0000 7f223b7fe700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f22340395b0 0x7f223403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.764+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f223c04da60 con 0x7f224c07f680 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.764+0000 7f2249d9b700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f22340395b0 0x7f223403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.765+0000 7f2249d9b700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f22340395b0 0x7f223403ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2240006fd0 tx=0x7f2240006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.769+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f223c019c70 con 0x7f224c07f680 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:09.987+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}) -- 0x7f224c105640 con 0x7f22340395b0 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.271+0000 7f223b7fe700 1 -- 172.21.15.1:0/1176713332 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f224c105640 con 0x7f22340395b0 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f22340395b0 msgr2=0x7f223403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f22340395b0 0x7f223403ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2240006fd0 tx=0x7f2240006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 msgr2=0x7f224c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c19eec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f223c011ac0 tx=0x7f223c011ba0 comp rx=0 tx=0).stop 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 shutdown_connections 2024-04-26T22:40:14.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f22340395b0 0x7f223403ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:14.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 --2- 172.21.15.1:0/1176713332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f224c07f680 0x7f224c19eec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:14.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 >> 172.21.15.1:0/1176713332 conn(0x7f224c104370 msgr2=0x7f224c104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:14.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 shutdown_connections 2024-04-26T22:40:14.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:14.274+0000 7f2250bce700 1 -- 172.21.15.1:0/1176713332 wait complete. 2024-04-26T22:40:14.806 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying unmanaged mon service... 2024-04-26T22:40:15.510 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:15 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:15.510 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:15 smithi001 ceph-mon[29435]: Added host smithi001 2024-04-26T22:40:15.510 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:15 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:16.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T22:40:16.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.274+0000 7f1d68f90700 1 Processor -- start 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.274+0000 7f1d68f90700 1 -- start start 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d68f90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6410bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d68f90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6407ce60 con 0x7f1d64109840 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6410bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6410bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52880/0 (socket says 172.21.15.1:52880) 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d6259c700 1 -- 172.21.15.1:0/3078415099 learned_addr learned my addr 172.21.15.1:0/3078415099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d6259c700 1 -- 172.21.15.1:0/3078415099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6410c170 con 0x7f1d64109840 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d6259c700 1 --2- 172.21.15.1:0/3078415099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6410bc30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d540098a0 tx=0x7f1d54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e661ab8e932c216 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d61d9b700 1 -- 172.21.15.1:0/3078415099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d54004160 con 0x7f1d64109840 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d61d9b700 1 -- 172.21.15.1:0/3078415099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1d54004300 con 0x7f1d64109840 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.275+0000 7f1d61d9b700 1 -- 172.21.15.1:0/3078415099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d54011af0 con 0x7f1d64109840 2024-04-26T22:40:16.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 -- 172.21.15.1:0/3078415099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 msgr2=0x7f1d6410bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 --2- 172.21.15.1:0/3078415099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6410bc30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d540098a0 tx=0x7f1d54009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 -- 172.21.15.1:0/3078415099 shutdown_connections 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 --2- 172.21.15.1:0/3078415099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6410bc30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 -- 172.21.15.1:0/3078415099 >> 172.21.15.1:0/3078415099 conn(0x7f1d641031f0 msgr2=0x7f1d64105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 -- 172.21.15.1:0/3078415099 shutdown_connections 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.276+0000 7f1d68f90700 1 -- 172.21.15.1:0/3078415099 wait complete. 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d68f90700 1 Processor -- start 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d68f90700 1 -- start start 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d68f90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d68f90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6410d130 con 0x7f1d64109840 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:16.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52890/0 (socket says 172.21.15.1:52890) 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d6259c700 1 -- 172.21.15.1:0/3099673811 learned_addr learned my addr 172.21.15.1:0/3099673811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.277+0000 7f1d6259c700 1 -- 172.21.15.1:0/3099673811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6419d220 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.278+0000 7f1d6259c700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6419cce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d54011ac0 tx=0x7f1d54011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.278+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d54003970 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.278+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6419d4b0 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.278+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1d54003b10 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.278+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d54024610 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.278+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d641a0190 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.279+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1d54024890 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.279+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d64064a80 con 0x7f1d64109840 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.279+0000 7f1d5bfff700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f1d500395b0 0x7f1d5003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:16.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.279+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1d5404da50 con 0x7f1d64109840 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.279+0000 7f1d59d9b700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f1d500395b0 0x7f1d5003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.279+0000 7f1d59d9b700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f1d500395b0 0x7f1d5003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c006fd0 tx=0x7f1d4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.284+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d5402d910 con 0x7f1d64109840 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.508+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1d64104430 con 0x7f1d500395b0 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.513+0000 7f1d5bfff700 1 -- 172.21.15.1:0/3099673811 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1d64104430 con 0x7f1d500395b0 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.515+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f1d500395b0 msgr2=0x7f1d5003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.515+0000 7f1d68f90700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f1d500395b0 0x7f1d5003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c006fd0 tx=0x7f1d4c006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.515+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 msgr2=0x7f1d6419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.515+0000 7f1d68f90700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6419cce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d54011ac0 tx=0x7f1d54011ba0 comp rx=0 tx=0).stop 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.516+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 shutdown_connections 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.516+0000 7f1d68f90700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f1d500395b0 0x7f1d5003ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.516+0000 7f1d68f90700 1 --2- 172.21.15.1:0/3099673811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d64109840 0x7f1d6419cce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.516+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 >> 172.21.15.1:0/3099673811 conn(0x7f1d641031f0 msgr2=0x7f1d64103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:16.380 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.516+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 shutdown_connections 2024-04-26T22:40:16.381 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:15.516+0000 7f1d68f90700 1 -- 172.21.15.1:0/3099673811 wait complete. 2024-04-26T22:40:16.381 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying unmanaged mgr service... 2024-04-26T22:40:16.714 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:16 smithi001 ceph-mon[29435]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:16.714 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:16 smithi001 ceph-mon[29435]: Saving service mon spec with placement count:5 2024-04-26T22:40:16.714 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:16 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:18.162 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:17 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:18.162 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:17 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:18.162 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:17 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.346+0000 7f5d557ad700 1 Processor -- start 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.346+0000 7f5d557ad700 1 -- start start 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.346+0000 7f5d557ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d5010bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.346+0000 7f5d557ad700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5007d730 con 0x7f5d501097c0 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d5010bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d5010bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52896/0 (socket says 172.21.15.1:52896) 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4effd700 1 -- 172.21.15.1:0/3203252290 learned_addr learned my addr 172.21.15.1:0/3203252290 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4effd700 1 -- 172.21.15.1:0/3203252290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5010c0f0 con 0x7f5d501097c0 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4effd700 1 --2- 172.21.15.1:0/3203252290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d5010bbb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d380098a0 tx=0x7f5d38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ae8be72e0c5c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4dffb700 1 -- 172.21.15.1:0/3203252290 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d38004160 con 0x7f5d501097c0 2024-04-26T22:40:18.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4dffb700 1 -- 172.21.15.1:0/3203252290 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d38004300 con 0x7f5d501097c0 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.347+0000 7f5d4dffb700 1 -- 172.21.15.1:0/3203252290 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d38011a20 con 0x7f5d501097c0 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 -- 172.21.15.1:0/3203252290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 msgr2=0x7f5d5010bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 --2- 172.21.15.1:0/3203252290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d5010bbb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d380098a0 tx=0x7f5d38009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 -- 172.21.15.1:0/3203252290 shutdown_connections 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 --2- 172.21.15.1:0/3203252290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d5010bbb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 -- 172.21.15.1:0/3203252290 >> 172.21.15.1:0/3203252290 conn(0x7f5d501031f0 msgr2=0x7f5d50105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 -- 172.21.15.1:0/3203252290 shutdown_connections 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.348+0000 7f5d557ad700 1 -- 172.21.15.1:0/3203252290 wait complete. 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d557ad700 1 Processor -- start 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d557ad700 1 -- start start 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d557ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d50109040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d557ad700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5010d100 con 0x7f5d501097c0 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d4effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d50109040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d4effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d50109040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52910/0 (socket says 172.21.15.1:52910) 2024-04-26T22:40:18.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d4effd700 1 -- 172.21.15.1:0/694384932 learned_addr learned my addr 172.21.15.1:0/694384932 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d4effd700 1 -- 172.21.15.1:0/694384932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d50109580 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.349+0000 7f5d4effd700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d50109040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d380116e0 tx=0x7f5d38011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.350+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d38003b20 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.350+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d3801b8f0 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.350+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d50107770 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.350+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d380238c0 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.350+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d50107cb0 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.351+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5d38023a60 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.351+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d50064a80 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.351+0000 7f5d47fff700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f5d3c0395b0 0x7f5d3c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.351+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5d3804e000 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.351+0000 7f5d4e7fc700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f5d3c0395b0 0x7f5d3c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.351+0000 7f5d4e7fc700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f5d3c0395b0 0x7f5d3c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006fd0 tx=0x7f5d40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.356+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5d3801f030 con 0x7f5d501097c0 2024-04-26T22:40:18.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.581+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5d50108490 con 0x7f5d3c0395b0 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.586+0000 7f5d47fff700 1 -- 172.21.15.1:0/694384932 <== mgr.14120 v2:172.21.15.1:6800/3267707627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5d50108490 con 0x7f5d3c0395b0 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f5d3c0395b0 msgr2=0x7f5d3c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f5d3c0395b0 0x7f5d3c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006fd0 tx=0x7f5d40006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 msgr2=0x7f5d50109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d50109040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d380116e0 tx=0x7f5d38011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 shutdown_connections 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f5d3c0395b0 0x7f5d3c03ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 --2- 172.21.15.1:0/694384932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d501097c0 0x7f5d50109040 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 >> 172.21.15.1:0/694384932 conn(0x7f5d501031f0 msgr2=0x7f5d50103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 shutdown_connections 2024-04-26T22:40:18.173 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:17.589+0000 7f5d557ad700 1 -- 172.21.15.1:0/694384932 wait complete. 2024-04-26T22:40:19.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:18 smithi001 ceph-mon[29435]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:19.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:18 smithi001 ceph-mon[29435]: Saving service mgr spec with placement count:2 2024-04-26T22:40:19.678 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bfaff9700 1 Processor -- start 2024-04-26T22:40:19.678 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bfaff9700 1 -- start start 2024-04-26T22:40:19.678 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bfaff9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf41091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:19.678 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bfaff9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf41097c0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.678 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bf8d95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf41091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:19.678 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bf8d95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf41091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52918/0 (socket says 172.21.15.1:52918) 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.763+0000 7f6bf8d95700 1 -- 172.21.15.1:0/573402035 learned_addr learned my addr 172.21.15.1:0/573402035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.764+0000 7f6bf8d95700 1 -- 172.21.15.1:0/573402035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf4109f30 con 0x7f6bf4108dd0 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.764+0000 7f6bf8d95700 1 --2- 172.21.15.1:0/573402035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf41091f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6be80098a0 tx=0x7f6be8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34259a8fd0f7166a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.764+0000 7f6bf37fe700 1 -- 172.21.15.1:0/573402035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be8004160 con 0x7f6bf4108dd0 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.764+0000 7f6bf37fe700 1 -- 172.21.15.1:0/573402035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6be8004300 con 0x7f6bf4108dd0 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.764+0000 7f6bf37fe700 1 -- 172.21.15.1:0/573402035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be80119d0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 -- 172.21.15.1:0/573402035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 msgr2=0x7f6bf41091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 --2- 172.21.15.1:0/573402035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf41091f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6be80098a0 tx=0x7f6be8009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 -- 172.21.15.1:0/573402035 shutdown_connections 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 --2- 172.21.15.1:0/573402035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf41091f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 -- 172.21.15.1:0/573402035 >> 172.21.15.1:0/573402035 conn(0x7f6bf4104370 msgr2=0x7f6bf41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 -- 172.21.15.1:0/573402035 shutdown_connections 2024-04-26T22:40:19.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.765+0000 7f6bfaff9700 1 -- 172.21.15.1:0/573402035 wait complete. 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bfaff9700 1 Processor -- start 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bfaff9700 1 -- start start 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bfaff9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bfaff9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf410d2e0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bf8d95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bf8d95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52926/0 (socket says 172.21.15.1:52926) 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bf8d95700 1 -- 172.21.15.1:0/402635757 learned_addr learned my addr 172.21.15.1:0/402635757 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.766+0000 7f6bf8d95700 1 -- 172.21.15.1:0/402635757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf419d160 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.767+0000 7f6bf8d95700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf419cc20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6be8006af0 tx=0x7f6be8003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.767+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be80119d0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.767+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6be80044b0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.767+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf419d3f0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.767+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be8023860 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.767+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf41a00e0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.768+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6be802c3f0 con 0x7f6bf4108dd0 2024-04-26T22:40:19.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.768+0000 7f6bf1ffb700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f6be40395b0 0x7f6be403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.768+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6be804ee80 con 0x7f6bf4108dd0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.768+0000 7f6bf3fff700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f6be40395b0 0x7f6be403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.768+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf4064a80 con 0x7f6bf4108dd0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.769+0000 7f6bf3fff700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f6be40395b0 0x7f6be403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6be0006fd0 tx=0x7f6be0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.774+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6be801f030 con 0x7f6bf4108dd0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:18.998+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f6bf41a0520 con 0x7f6bf4108dd0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.001+0000 7f6bf1ffb700 1 -- 172.21.15.1:0/402635757 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f6be802b760 con 0x7f6bf4108dd0 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f6be40395b0 msgr2=0x7f6be403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f6be40395b0 0x7f6be403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6be0006fd0 tx=0x7f6be0006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 msgr2=0x7f6bf419cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf419cc20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6be8006af0 tx=0x7f6be8003980 comp rx=0 tx=0).stop 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 shutdown_connections 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f6be40395b0 0x7f6be403ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 --2- 172.21.15.1:0/402635757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf4108dd0 0x7f6bf419cc20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:19.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 >> 172.21.15.1:0/402635757 conn(0x7f6bf4104370 msgr2=0x7f6bf4104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:19.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 shutdown_connections 2024-04-26T22:40:19.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:19.003+0000 7f6bfaff9700 1 -- 172.21.15.1:0/402635757 wait complete. 2024-04-26T22:40:20.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:20 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/402635757' entity='client.admin' 2024-04-26T22:40:21.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.273+0000 7f89147bc700 1 Processor -- start 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.273+0000 7f89147bc700 1 -- start start 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f89147bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c1080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f89147bc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f890c108680 con 0x7f890c107c90 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8912558700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c1080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8912558700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c1080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52930/0 (socket says 172.21.15.1:52930) 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8912558700 1 -- 172.21.15.1:0/1681268444 learned_addr learned my addr 172.21.15.1:0/1681268444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8912558700 1 -- 172.21.15.1:0/1681268444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890c108e40 con 0x7f890c107c90 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8912558700 1 --2- 172.21.15.1:0/1681268444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c1080b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f890800bb40 tx=0x7f890800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c890a043fbf4cda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8911556700 1 -- 172.21.15.1:0/1681268444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f890800dbe0 con 0x7f890c107c90 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8911556700 1 -- 172.21.15.1:0/1681268444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89080044e0 con 0x7f890c107c90 2024-04-26T22:40:21.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.274+0000 7f8911556700 1 -- 172.21.15.1:0/1681268444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908010470 con 0x7f890c107c90 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 -- 172.21.15.1:0/1681268444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 msgr2=0x7f890c1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 --2- 172.21.15.1:0/1681268444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c1080b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f890800bb40 tx=0x7f890800be50 comp rx=0 tx=0).stop 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 -- 172.21.15.1:0/1681268444 shutdown_connections 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 --2- 172.21.15.1:0/1681268444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c1080b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 -- 172.21.15.1:0/1681268444 >> 172.21.15.1:0/1681268444 conn(0x7f890c103230 msgr2=0x7f890c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 -- 172.21.15.1:0/1681268444 shutdown_connections 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.275+0000 7f89147bc700 1 -- 172.21.15.1:0/1681268444 wait complete. 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f89147bc700 1 Processor -- start 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f89147bc700 1 -- start start 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f89147bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f89147bc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f890c07c850 con 0x7f890c107c90 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f8912558700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:21.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f8912558700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c19aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52936/0 (socket says 172.21.15.1:52936) 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f8912558700 1 -- 172.21.15.1:0/124868053 learned_addr learned my addr 172.21.15.1:0/124868053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f8912558700 1 -- 172.21.15.1:0/124868053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890c19afc0 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.276+0000 7f8912558700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c19aa80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8908003c30 tx=0x7f8908003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.277+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908027070 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.277+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8908010bd0 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.277+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890c19b250 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.277+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f890c19df40 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.277+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908022a60 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.277+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f890802a070 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.278+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f890c064a80 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.278+0000 7f89037fe700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f88f80395b0 0x7f88f803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.278+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f890801f070 con 0x7f890c107c90 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.278+0000 7f8911d57700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f88f80395b0 0x7f88f803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:21.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.278+0000 7f8911d57700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f88f80395b0 0x7f88f803ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006fd0 tx=0x7f88fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.283+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8908015070 con 0x7f890c107c90 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.492+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f890c193fe0 con 0x7f890c107c90 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.494+0000 7f89037fe700 1 -- 172.21.15.1:0/124868053 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f890c193fe0 con 0x7f890c107c90 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f88f80395b0 msgr2=0x7f88f803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f88f80395b0 0x7f88f803ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006fd0 tx=0x7f88fc006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 msgr2=0x7f890c19aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c19aa80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8908003c30 tx=0x7f8908003ff0 comp rx=0 tx=0).stop 2024-04-26T22:40:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 shutdown_connections 2024-04-26T22:40:21.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f88f80395b0 0x7f88f803ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:21.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 --2- 172.21.15.1:0/124868053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c107c90 0x7f890c19aa80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:21.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 >> 172.21.15.1:0/124868053 conn(0x7f890c103230 msgr2=0x7f890c1910c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:21.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 shutdown_connections 2024-04-26T22:40:21.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:20.496+0000 7f89147bc700 1 -- 172.21.15.1:0/124868053 wait complete. 2024-04-26T22:40:21.108 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling the dashboard module... 2024-04-26T22:40:21.637 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:21 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/124868053' entity='client.admin' 2024-04-26T22:40:21.637 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:21 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:22.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/401414369' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T22:40:22.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-mon[29435]: from='mgr.14120 172.21.15.1:0/3268592173' entity='mgr.x' 2024-04-26T22:40:22.810 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: ignoring --setuser ceph since I am not root 2024-04-26T22:40:22.811 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: ignoring --setgroup ceph since I am not root 2024-04-26T22:40:22.811 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:22.679+0000 7fd49a6de200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T22:40:23.093 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:22.873+0000 7fd49a6de200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:40:23.093 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:22.979+0000 7fd49a6de200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.551+0000 7fe3b9780700 1 Processor -- start 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.551+0000 7fe3b9780700 1 -- start start 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.551+0000 7fe3b9780700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b410bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.551+0000 7fe3b9780700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b407d6d0 con 0x7fe3b4109840 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b410bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b410bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52948/0 (socket says 172.21.15.1:52948) 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b2ffd700 1 -- 172.21.15.1:0/486050909 learned_addr learned my addr 172.21.15.1:0/486050909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b2ffd700 1 -- 172.21.15.1:0/486050909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b410c170 con 0x7fe3b4109840 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b2ffd700 1 --2- 172.21.15.1:0/486050909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b410bc30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40098a0 tx=0x7fe3a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7b8644db6ad0e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b1ffb700 1 -- 172.21.15.1:0/486050909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a4004160 con 0x7fe3b4109840 2024-04-26T22:40:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b1ffb700 1 -- 172.21.15.1:0/486050909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3a4004300 con 0x7fe3b4109840 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.552+0000 7fe3b1ffb700 1 -- 172.21.15.1:0/486050909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a4011a20 con 0x7fe3b4109840 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 -- 172.21.15.1:0/486050909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 msgr2=0x7fe3b410bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 --2- 172.21.15.1:0/486050909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b410bc30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40098a0 tx=0x7fe3a4009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 -- 172.21.15.1:0/486050909 shutdown_connections 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 --2- 172.21.15.1:0/486050909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b410bc30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 -- 172.21.15.1:0/486050909 >> 172.21.15.1:0/486050909 conn(0x7fe3b41031d0 msgr2=0x7fe3b4105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 -- 172.21.15.1:0/486050909 shutdown_connections 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.553+0000 7fe3b9780700 1 -- 172.21.15.1:0/486050909 wait complete. 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b9780700 1 Processor -- start 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b9780700 1 -- start start 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b9780700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b41090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b9780700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b410d120 con 0x7fe3b4109840 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b41090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b41090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52950/0 (socket says 172.21.15.1:52950) 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b2ffd700 1 -- 172.21.15.1:0/401414369 learned_addr learned my addr 172.21.15.1:0/401414369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.554+0000 7fe3b2ffd700 1 -- 172.21.15.1:0/401414369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b4109600 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.555+0000 7fe3b2ffd700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b41090c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4004660 tx=0x7fe3a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.555+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a4003aa0 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.555+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b41077f0 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.555+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3a401b8f0 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.555+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a4023900 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.555+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b4107d30 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.556+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe3a4023bc0 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.556+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b4064a80 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.556+0000 7fe39bfff700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fe39c0395b0 0x7fe39c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.556+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe3a404df40 con 0x7fe3b4109840 2024-04-26T22:40:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.556+0000 7fe3b27fc700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fe39c0395b0 0x7fe39c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.557+0000 7fe3b27fc700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fe39c0395b0 0x7fe39c03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8006fd0 tx=0x7fe3a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.565+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe3a401f030 con 0x7fe3b4109840 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:21.841+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fe3b4108610 con 0x7fe3b4109840 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.496+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fe3a4010380 con 0x7fe3b4109840 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.498+0000 7fe39bfff700 1 -- 172.21.15.1:0/401414369 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fe3a401baa0 con 0x7fe3b4109840 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fe39c0395b0 msgr2=0x7fe39c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fe39c0395b0 0x7fe39c03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8006fd0 tx=0x7fe3a8006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 msgr2=0x7fe3b41090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b41090c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4004660 tx=0x7fe3a4011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 shutdown_connections 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7fe39c0395b0 0x7fe39c03ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 --2- 172.21.15.1:0/401414369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3b4109840 0x7fe3b41090c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 >> 172.21.15.1:0/401414369 conn(0x7fe3b41031d0 msgr2=0x7fe3b4103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 shutdown_connections 2024-04-26T22:40:23.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:22.500+0000 7fe3b9780700 1 -- 172.21.15.1:0/401414369 wait complete. 2024-04-26T22:40:23.346 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:23 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:23.304+0000 7fd49a6de200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T22:40:23.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:23 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/401414369' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T22:40:23.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:23 smithi001 ceph-mon[29435]: mgrmap e9: x(active, since 19s) 2024-04-26T22:40:23.749 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:23 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:23.395+0000 7fd49a6de200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T22:40:24.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.541+0000 7f08c6310700 1 Processor -- start 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.541+0000 7f08c6310700 1 -- start start 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08c6310700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c007db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08c6310700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f08c007e130 con 0x7f08c007f700 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c007db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c007db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44796/0 (socket says 172.21.15.1:44796) 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08bffff700 1 -- 172.21.15.1:0/845785439 learned_addr learned my addr 172.21.15.1:0/845785439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:24.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08bffff700 1 -- 172.21.15.1:0/845785439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c007e2b0 con 0x7f08c007f700 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08bffff700 1 --2- 172.21.15.1:0/845785439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c007db60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08b00098a0 tx=0x7f08b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a3eb882eeb625b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08beffd700 1 -- 172.21.15.1:0/845785439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b0004160 con 0x7f08c007f700 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08beffd700 1 -- 172.21.15.1:0/845785439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08b0004300 con 0x7f08c007f700 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.542+0000 7f08beffd700 1 -- 172.21.15.1:0/845785439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b0011a20 con 0x7f08c007f700 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 -- 172.21.15.1:0/845785439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 msgr2=0x7f08c007db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 --2- 172.21.15.1:0/845785439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c007db60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08b00098a0 tx=0x7f08b0009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 -- 172.21.15.1:0/845785439 shutdown_connections 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 --2- 172.21.15.1:0/845785439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c007db60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 -- 172.21.15.1:0/845785439 >> 172.21.15.1:0/845785439 conn(0x7f08c0104310 msgr2=0x7f08c0106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 -- 172.21.15.1:0/845785439 shutdown_connections 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.543+0000 7f08c6310700 1 -- 172.21.15.1:0/845785439 wait complete. 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08c6310700 1 Processor -- start 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08c6310700 1 -- start start 2024-04-26T22:40:24.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08c6310700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08c6310700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f08c010f390 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c019eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44798/0 (socket says 172.21.15.1:44798) 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08bffff700 1 -- 172.21.15.1:0/3933769312 learned_addr learned my addr 172.21.15.1:0/3933769312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.544+0000 7f08bffff700 1 -- 172.21.15.1:0/3933769312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c019f410 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.545+0000 7f08bffff700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c019eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f08b00116e0 tx=0x7f08b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.545+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b0003b20 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.545+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08b001b8f0 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.545+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c019f6a0 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.545+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08b00238c0 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.545+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c01a2340 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.546+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f08b0023a60 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.546+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c0064a80 con 0x7f08c007f700 2024-04-26T22:40:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.546+0000 7f08bd7fa700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 0x7f08a803bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.546+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f08b004e090 con 0x7f08c007f700 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.546+0000 7f08bf7fe700 1 -- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 msgr2=0x7f08a803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.546+0000 7f08bf7fe700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 0x7f08a803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.551+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f08b001f030 con 0x7f08c007f700 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.747+0000 7f08bf7fe700 1 -- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 msgr2=0x7f08a803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.747+0000 7f08bf7fe700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 0x7f08a803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.862+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f08c01a28f0 con 0x7f08c007f700 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.862+0000 7f08bd7fa700 1 -- 172.21.15.1:0/3933769312 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f08b0020d70 con 0x7f08c007f700 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.864+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 msgr2=0x7f08a803bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.864+0000 7f08c6310700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 0x7f08a803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.864+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 msgr2=0x7f08c019eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.864+0000 7f08c6310700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c019eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f08b00116e0 tx=0x7f08b0011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:24.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.864+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 shutdown_connections 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.865+0000 7f08c6310700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f08a8039610 0x7f08a803bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.865+0000 7f08c6310700 1 --2- 172.21.15.1:0/3933769312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08c007f700 0x7f08c019eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.865+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 >> 172.21.15.1:0/3933769312 conn(0x7f08c0104310 msgr2=0x7f08c0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.865+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 shutdown_connections 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:23.865+0000 7f08c6310700 1 -- 172.21.15.1:0/3933769312 wait complete. 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-04-26T22:40:24.413 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 9... 2024-04-26T22:40:24.761 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:24 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3933769312' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T22:40:24.761 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:24 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:24.660+0000 7fd49a6de200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T22:40:25.020 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:24 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:24.841+0000 7fd49a6de200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T22:40:25.021 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:25.019+0000 7fd49a6de200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T22:40:25.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:25.320+0000 7fd49a6de200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T22:40:25.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:25.417+0000 7fd49a6de200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T22:40:26.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:25.628+0000 7fd49a6de200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T22:40:26.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:25.829+0000 7fd49a6de200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:40:26.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:26 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:26.458+0000 7fd49a6de200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T22:40:27.216 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:26 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:26.903+0000 7fd49a6de200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T22:40:27.217 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:27 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:27.004+0000 7fd49a6de200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T22:40:27.217 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:27 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:27.216+0000 7fd49a6de200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T22:40:28.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:27 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:27.784+0000 7fd49a6de200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T22:40:28.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:27 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:27.881+0000 7fd49a6de200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T22:40:28.496 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:28.137+0000 7fd49a6de200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T22:40:28.496 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:28.276+0000 7fd49a6de200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T22:40:28.496 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:28.375+0000 7fd49a6de200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T22:40:28.496 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:28.495+0000 7fd49a6de200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T22:40:29.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:28.806+0000 7fd49a6de200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T22:40:29.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:29.437+0000 7fd49a6de200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T22:40:30.096 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:40:29.663+0000 7fd49a6de200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T22:40:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: Active manager daemon x restarted 2024-04-26T22:40:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: Activating manager daemon x 2024-04-26T22:40:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: mgrmap e10: x(active, starting, since 0.00343524s) 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: Manager daemon x is now available 2024-04-26T22:40:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:30.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T22:40:30.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T22:40:30.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:29] ENGINE Bus STARTING 2024-04-26T22:40:30.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:30] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T22:40:30.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:30] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T22:40:30.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: [26/Apr/2024:22:40:30] ENGINE Bus STARTED 2024-04-26T22:40:30.967 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:30 smithi001 ceph-mon[29435]: mgrmap e11: x(active, since 1.00615s) 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.863+0000 7f8b85476700 1 Processor -- start 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.863+0000 7f8b85476700 1 -- start start 2024-04-26T22:40:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.863+0000 7f8b85476700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b80109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.863+0000 7f8b85476700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8b80109760 con 0x7f8b80108d70 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.863+0000 7f8b7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b80109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b80109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44804/0 (socket says 172.21.15.1:44804) 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7effd700 1 -- 172.21.15.1:0/4072063691 learned_addr learned my addr 172.21.15.1:0/4072063691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7effd700 1 -- 172.21.15.1:0/4072063691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b80109f70 con 0x7f8b80108d70 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7effd700 1 --2- 172.21.15.1:0/4072063691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b80109190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8b740098a0 tx=0x7f8b74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c44149e223b8060 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7dffb700 1 -- 172.21.15.1:0/4072063691 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b74004160 con 0x7f8b80108d70 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7dffb700 1 -- 172.21.15.1:0/4072063691 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b74004300 con 0x7f8b80108d70 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.864+0000 7f8b7dffb700 1 -- 172.21.15.1:0/4072063691 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b74011a20 con 0x7f8b80108d70 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 -- 172.21.15.1:0/4072063691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 msgr2=0x7f8b80109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 --2- 172.21.15.1:0/4072063691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b80109190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8b740098a0 tx=0x7f8b74009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 -- 172.21.15.1:0/4072063691 shutdown_connections 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 --2- 172.21.15.1:0/4072063691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b80109190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 -- 172.21.15.1:0/4072063691 >> 172.21.15.1:0/4072063691 conn(0x7f8b801042d0 msgr2=0x7f8b80106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 -- 172.21.15.1:0/4072063691 shutdown_connections 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.865+0000 7f8b85476700 1 -- 172.21.15.1:0/4072063691 wait complete. 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b85476700 1 Processor -- start 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b85476700 1 -- start start 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b85476700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b8007eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b85476700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8010d320 con 0x7f8b80108d70 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b8007eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b8007eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44812/0 (socket says 172.21.15.1:44812) 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b7effd700 1 -- 172.21.15.1:0/363971336 learned_addr learned my addr 172.21.15.1:0/363971336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.866+0000 7f8b7effd700 1 -- 172.21.15.1:0/363971336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8007f0a0 con 0x7f8b80108d70 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.867+0000 7f8b7effd700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b8007eb60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b740119f0 tx=0x7f8b74011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:31.212 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.867+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b740039e0 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.867+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b7401b8f0 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.867+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8007d290 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.867+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b740237a0 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.867+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8007d7d0 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.868+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f8b7402c3f0 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.868+0000 7f8b67fff700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.868+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 --> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8b6803c1e0 con 0x7f8b68039610 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.868+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8b7404dfe0 con 0x7f8b80108d70 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.868+0000 7f8b7e7fc700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 msgr2=0x7f8b6803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:24.868+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:25.069+0000 7f8b7e7fc700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 msgr2=0x7f8b6803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:25.069+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:25.470+0000 7f8b7e7fc700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 msgr2=0x7f8b6803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:31.213 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:25.470+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:26.271+0000 7f8b7e7fc700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 msgr2=0x7f8b6803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:26.271+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:27.873+0000 7f8b7e7fc700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 msgr2=0x7f8b6803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3267707627 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:27.873+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:29.667+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f8b7402c780 con 0x7f8b80108d70 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:29.667+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 msgr2=0x7f8b6803bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:29.667+0000 7f8b67fff700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.669+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f8b7404fe40 con 0x7f8b80108d70 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.669+0000 7f8b67fff700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b6803cff0 0x7f8b6803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.669+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8b740105a0 con 0x7f8b6803cff0 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.670+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b6803cff0 0x7f8b6803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.670+0000 7f8b7e7fc700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b6803cff0 0x7f8b6803f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8b70003a10 tx=0x7f8b70009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.671+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f8b740105a0 con 0x7f8b6803cff0 2024-04-26T22:40:31.214 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.679+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8b8006a540 con 0x7f8b6803cff0 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.679+0000 7f8b67fff700 1 -- 172.21.15.1:0/363971336 <== mgr.14152 v2:172.21.15.1:6800/4125558078 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f8b8006a540 con 0x7f8b6803cff0 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.679+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b6803cff0 msgr2=0x7f8b6803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.679+0000 7f8b85476700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b6803cff0 0x7f8b6803f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8b70003a10 tx=0x7f8b70009270 comp rx=0 tx=0).stop 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.679+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 msgr2=0x7f8b8007eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.679+0000 7f8b85476700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b8007eb60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b740119f0 tx=0x7f8b74011ad0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 shutdown_connections 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b6803cff0 0x7f8b6803f3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b80108d70 0x7f8b8007eb60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.215 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 --2- 172.21.15.1:0/363971336 >> [v2:172.21.15.1:6800/3267707627,v1:172.21.15.1:6801/3267707627] conn(0x7f8b68039610 0x7f8b6803bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 >> 172.21.15.1:0/363971336 conn(0x7f8b801042d0 msgr2=0x7f8b80104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 shutdown_connections 2024-04-26T22:40:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:30.680+0000 7f8b85476700 1 -- 172.21.15.1:0/363971336 wait complete. 2024-04-26T22:40:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 9 is available 2024-04-26T22:40:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:Generating a dashboard self-signed certificate... 2024-04-26T22:40:31.979 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:31 smithi001 ceph-mon[29435]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T22:40:31.979 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:31 smithi001 ceph-mon[29435]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff988f18700 1 Processor -- start 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff988f18700 1 -- start start 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff988f18700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984107490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff988f18700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff984107a60 con 0x7ff984107070 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff98259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff98259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48838/0 (socket says 172.21.15.1:48838) 2024-04-26T22:40:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.984+0000 7ff98259c700 1 -- 172.21.15.1:0/3702790115 learned_addr learned my addr 172.21.15.1:0/3702790115 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.985+0000 7ff98259c700 1 -- 172.21.15.1:0/3702790115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff984107be0 con 0x7ff984107070 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.985+0000 7ff98259c700 1 --2- 172.21.15.1:0/3702790115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984107490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9740098a0 tx=0x7ff974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2808be460c2286f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.985+0000 7ff98159a700 1 -- 172.21.15.1:0/3702790115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff974004160 con 0x7ff984107070 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.985+0000 7ff98159a700 1 -- 172.21.15.1:0/3702790115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff974004300 con 0x7ff984107070 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.985+0000 7ff98159a700 1 -- 172.21.15.1:0/3702790115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff974011a20 con 0x7ff984107070 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 -- 172.21.15.1:0/3702790115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 msgr2=0x7ff984107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 --2- 172.21.15.1:0/3702790115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984107490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9740098a0 tx=0x7ff974009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 -- 172.21.15.1:0/3702790115 shutdown_connections 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 --2- 172.21.15.1:0/3702790115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984107490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 -- 172.21.15.1:0/3702790115 >> 172.21.15.1:0/3702790115 conn(0x7ff984102970 msgr2=0x7ff984104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 -- 172.21.15.1:0/3702790115 shutdown_connections 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.986+0000 7ff988f18700 1 -- 172.21.15.1:0/3702790115 wait complete. 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff988f18700 1 Processor -- start 2024-04-26T22:40:32.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff988f18700 1 -- start start 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff988f18700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984109970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff988f18700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410af80 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff98259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984109970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff98259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984109970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48854/0 (socket says 172.21.15.1:48854) 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff98259c700 1 -- 172.21.15.1:0/1665317707 learned_addr learned my addr 172.21.15.1:0/1665317707 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff98259c700 1 -- 172.21.15.1:0/1665317707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff984109eb0 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff98259c700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984109970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff974000c00 tx=0x7ff974011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.987+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff974003aa0 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.988+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97401b8f0 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.988+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9841080a0 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.988+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff974023890 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.988+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9841085e0 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.988+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff97402c3f0 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.988+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff984064a80 con 0x7ff984107070 2024-04-26T22:40:32.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.989+0000 7ff9737fe700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff96c039600 0x7ff96c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.989+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff97404df60 con 0x7ff984107070 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.989+0000 7ff981d9b700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff96c039600 0x7ff96c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.989+0000 7ff981d9b700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff96c039600 0x7ff96c03bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff978006fd0 tx=0x7ff978006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:31.994+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff97401f030 con 0x7ff984107070 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.207+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff984108eb0 con 0x7ff96c039600 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.342+0000 7ff9737fe700 1 -- 172.21.15.1:0/1665317707 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff984108eb0 con 0x7ff96c039600 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff96c039600 msgr2=0x7ff96c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff96c039600 0x7ff96c03bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff978006fd0 tx=0x7ff978006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 msgr2=0x7ff984109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984109970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff974000c00 tx=0x7ff974011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 shutdown_connections 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff96c039600 0x7ff96c03bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:32.902 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 --2- 172.21.15.1:0/1665317707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff984107070 0x7ff984109970 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:32.903 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 >> 172.21.15.1:0/1665317707 conn(0x7ff984102970 msgr2=0x7ff984103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:32.903 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 shutdown_connections 2024-04-26T22:40:32.903 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:32.344+0000 7ff988f18700 1 -- 172.21.15.1:0/1665317707 wait complete. 2024-04-26T22:40:32.903 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial admin user... 2024-04-26T22:40:33.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:32 smithi001 ceph-mon[29435]: mgrmap e12: x(active, since 2s) 2024-04-26T22:40:33.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:33.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:32 smithi001 ceph-mon[29435]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:33.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:33.019 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:34.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:34.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$r7Ks5P8nkSSrcRrkNNotB.f8jWBVvYbXYu.rXllLS6XiEKUxCwVju", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714171233, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd90b8dd700 1 Processor -- start 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd90b8dd700 1 -- start start 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd90b8dd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90407db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd90b8dd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd90407e130 con 0x7fd90407f700 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd909679700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd909679700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48870/0 (socket says 172.21.15.1:48870) 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.410+0000 7fd909679700 1 -- 172.21.15.1:0/2464223926 learned_addr learned my addr 172.21.15.1:0/2464223926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.411+0000 7fd909679700 1 -- 172.21.15.1:0/2464223926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90407e2b0 con 0x7fd90407f700 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.411+0000 7fd909679700 1 --2- 172.21.15.1:0/2464223926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90407db60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098a0 tx=0x7fd900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d038216ce96111f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.411+0000 7fd8fbfff700 1 -- 172.21.15.1:0/2464223926 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd900004160 con 0x7fd90407f700 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.411+0000 7fd8fbfff700 1 -- 172.21.15.1:0/2464223926 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd900004300 con 0x7fd90407f700 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.411+0000 7fd8fbfff700 1 -- 172.21.15.1:0/2464223926 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd900011a20 con 0x7fd90407f700 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2464223926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 msgr2=0x7fd90407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 --2- 172.21.15.1:0/2464223926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90407db60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098a0 tx=0x7fd900009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2464223926 shutdown_connections 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 --2- 172.21.15.1:0/2464223926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90407db60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2464223926 >> 172.21.15.1:0/2464223926 conn(0x7fd904104310 msgr2=0x7fd904106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2464223926 shutdown_connections 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.412+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2464223926 wait complete. 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd90b8dd700 1 Processor -- start 2024-04-26T22:40:34.685 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd90b8dd700 1 -- start start 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd90b8dd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd90b8dd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd90410f370 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd909679700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd909679700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48874/0 (socket says 172.21.15.1:48874) 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd909679700 1 -- 172.21.15.1:0/2789847019 learned_addr learned my addr 172.21.15.1:0/2789847019 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.413+0000 7fd909679700 1 -- 172.21.15.1:0/2789847019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90419f360 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.414+0000 7fd909679700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90419ee20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9000116e0 tx=0x7fd900011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.414+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd900003b20 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.414+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd90001b8f0 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.414+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90419f5f0 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.414+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9000238c0 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.414+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9041a22e0 con 0x7fd90407f700 2024-04-26T22:40:34.686 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.415+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd904064a80 con 0x7fd90407f700 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.415+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fd90001baa0 con 0x7fd90407f700 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.415+0000 7fd8fa7fc700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd8f00395b0 0x7fd8f003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.415+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd90004e170 con 0x7fd90407f700 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.416+0000 7fd908e78700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd8f00395b0 0x7fd8f003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.416+0000 7fd908e78700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd8f00395b0 0x7fd8f003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4006fd0 tx=0x7fd8f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.421+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd90001f030 con 0x7fd90407f700 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.657+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd90419fad0 con 0x7fd8f00395b0 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.904+0000 7fd8fa7fc700 1 -- 172.21.15.1:0/2789847019 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd90419fad0 con 0x7fd8f00395b0 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd8f00395b0 msgr2=0x7fd8f003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd8f00395b0 0x7fd8f003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4006fd0 tx=0x7fd8f4006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 msgr2=0x7fd90419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90419ee20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9000116e0 tx=0x7fd900011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 shutdown_connections 2024-04-26T22:40:34.688 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd8f00395b0 0x7fd8f003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:34.688 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 --2- 172.21.15.1:0/2789847019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd90407f700 0x7fd90419ee20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:34.688 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 >> 172.21.15.1:0/2789847019 conn(0x7fd904104310 msgr2=0x7fd904104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:34.688 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 shutdown_connections 2024-04-26T22:40:34.688 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:33.906+0000 7fd90b8dd700 1 -- 172.21.15.1:0/2789847019 wait complete. 2024-04-26T22:40:34.688 INFO:teuthology.orchestra.run.smithi001.stdout:Fetching dashboard port number... 2024-04-26T22:40:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:34 smithi001 ceph-mon[29435]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:36.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:35 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3120484821' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.155+0000 7f29b602a700 1 Processor -- start 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.155+0000 7f29b602a700 1 -- start start 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29b602a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b010bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29b602a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29b007cea0 con 0x7f29b0109800 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29af7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b010bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29af7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b010bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48878/0 (socket says 172.21.15.1:48878) 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29af7fe700 1 -- 172.21.15.1:0/880001107 learned_addr learned my addr 172.21.15.1:0/880001107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29af7fe700 1 -- 172.21.15.1:0/880001107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b010c130 con 0x7f29b0109800 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.156+0000 7f29af7fe700 1 --2- 172.21.15.1:0/880001107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b010bbf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29a00098a0 tx=0x7f29a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a06214328de3cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29ae7fc700 1 -- 172.21.15.1:0/880001107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29a0004160 con 0x7f29b0109800 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29ae7fc700 1 -- 172.21.15.1:0/880001107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29a0004300 con 0x7f29b0109800 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29ae7fc700 1 -- 172.21.15.1:0/880001107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29a0011a20 con 0x7f29b0109800 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 -- 172.21.15.1:0/880001107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 msgr2=0x7f29b010bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 --2- 172.21.15.1:0/880001107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b010bbf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29a00098a0 tx=0x7f29a0009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 -- 172.21.15.1:0/880001107 shutdown_connections 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 --2- 172.21.15.1:0/880001107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b010bbf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 -- 172.21.15.1:0/880001107 >> 172.21.15.1:0/880001107 conn(0x7f29b01031f0 msgr2=0x7f29b0105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 -- 172.21.15.1:0/880001107 shutdown_connections 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.157+0000 7f29b602a700 1 -- 172.21.15.1:0/880001107 wait complete. 2024-04-26T22:40:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29b602a700 1 Processor -- start 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29b602a700 1 -- start start 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29b602a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29b602a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29b010d140 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29af7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29af7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48888/0 (socket says 172.21.15.1:48888) 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.158+0000 7f29af7fe700 1 -- 172.21.15.1:0/3120484821 learned_addr learned my addr 172.21.15.1:0/3120484821 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29af7fe700 1 -- 172.21.15.1:0/3120484821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b019d1e0 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29af7fe700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b019cca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f29a0004660 tx=0x7f29a0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29a0003aa0 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29a001b8f0 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b019d470 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29a0023900 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.159+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b01a0160 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.160+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f29a0023b70 con 0x7f29b0109800 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.160+0000 7f29acff9700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2998039660 0x7f299803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:36.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.160+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f29a004e150 con 0x7f29b0109800 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.160+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b0064a80 con 0x7f29b0109800 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.160+0000 7f29aeffd700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2998039660 0x7f299803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.161+0000 7f29aeffd700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2998039660 0x7f299803bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f29a4006fd0 tx=0x7f29a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.165+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f29a001f030 con 0x7f29b0109800 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.376+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f29b01a0670 con 0x7f29b0109800 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.376+0000 7f29acff9700 1 -- 172.21.15.1:0/3120484821 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f29a0010380 con 0x7f29b0109800 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.378+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2998039660 msgr2=0x7f299803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.378+0000 7f29b602a700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2998039660 0x7f299803bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f29a4006fd0 tx=0x7f29a4006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.378+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 msgr2=0x7f29b019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.378+0000 7f29b602a700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b019cca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f29a0004660 tx=0x7f29a0011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.379+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 shutdown_connections 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.379+0000 7f29b602a700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2998039660 0x7f299803bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.379+0000 7f29b602a700 1 --2- 172.21.15.1:0/3120484821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b0109800 0x7f29b019cca0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.379+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 >> 172.21.15.1:0/3120484821 conn(0x7f29b01031f0 msgr2=0x7f29b0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:36.112 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.379+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 shutdown_connections 2024-04-26T22:40:36.113 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:35.379+0000 7f29b602a700 1 -- 172.21.15.1:0/3120484821 wait complete. 2024-04-26T22:40:36.117 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T22:40:36.117 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T22:40:36.125 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T22:40:36.125 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T22:40:36.125 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T22:40:36.125 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T22:40:36.127 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph Dashboard is now available at: 2024-04-26T22:40:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: URL: https://smithi001:8443/ 2024-04-26T22:40:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: User: admin 2024-04-26T22:40:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: Password: m6smzjzi1w 2024-04-26T22:40:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:36.128 INFO:teuthology.orchestra.run.smithi001.stdout:Saving cluster configuration to /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config directory 2024-04-26T22:40:36.128 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling autotune for osd_memory_target 2024-04-26T22:40:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:36 smithi001 ceph-mon[29435]: mgrmap e13: x(active, since 6s) 2024-04-26T22:40:37.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.575+0000 7f00d46af700 1 Processor -- start 2024-04-26T22:40:37.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.575+0000 7f00d46af700 1 -- start start 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d46af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d46af700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc1097c0 con 0x7f00cc108dd0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d244b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d244b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48902/0 (socket says 172.21.15.1:48902) 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d244b700 1 -- 172.21.15.1:0/327586901 learned_addr learned my addr 172.21.15.1:0/327586901 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d244b700 1 -- 172.21.15.1:0/327586901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc109f30 con 0x7f00cc108dd0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.576+0000 7f00d244b700 1 --2- 172.21.15.1:0/327586901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc1091f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00c80098a0 tx=0x7f00c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb39185435f7c7b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d1449700 1 -- 172.21.15.1:0/327586901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00c8004160 con 0x7f00cc108dd0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d1449700 1 -- 172.21.15.1:0/327586901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00c8004300 con 0x7f00cc108dd0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d1449700 1 -- 172.21.15.1:0/327586901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00c8011a20 con 0x7f00cc108dd0 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 -- 172.21.15.1:0/327586901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 msgr2=0x7f00cc1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:37.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 --2- 172.21.15.1:0/327586901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc1091f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00c80098a0 tx=0x7f00c8009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 -- 172.21.15.1:0/327586901 shutdown_connections 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 --2- 172.21.15.1:0/327586901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc1091f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 -- 172.21.15.1:0/327586901 >> 172.21.15.1:0/327586901 conn(0x7f00cc104370 msgr2=0x7f00cc1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 -- 172.21.15.1:0/327586901 shutdown_connections 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.577+0000 7f00d46af700 1 -- 172.21.15.1:0/327586901 wait complete. 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d46af700 1 Processor -- start 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d46af700 1 -- start start 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d46af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d46af700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc10d2e0 con 0x7f00cc108dd0 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d244b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d244b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc19cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48908/0 (socket says 172.21.15.1:48908) 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.578+0000 7f00d244b700 1 -- 172.21.15.1:0/2884812546 learned_addr learned my addr 172.21.15.1:0/2884812546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00d244b700 1 -- 172.21.15.1:0/2884812546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc19d1c0 con 0x7f00cc108dd0 2024-04-26T22:40:37.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00d244b700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc19cc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00c8000c00 tx=0x7f00c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00c8003aa0 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00c801b8f0 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc19d450 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00c8023890 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.579+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc1a0140 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.580+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f00c802c3f0 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.580+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00cc064a80 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.580+0000 7f00c37fe700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f00b80396a0 0x7f00b803bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.580+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f00c804e0a0 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.580+0000 7f00d1c4a700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f00b80396a0 0x7f00b803bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.581+0000 7f00d1c4a700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f00b80396a0 0x7f00b803bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f00bc006fd0 tx=0x7f00bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.585+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f00c801f030 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.791+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f00cc07dea0 con 0x7f00cc108dd0 2024-04-26T22:40:37.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.791+0000 7f00c37fe700 1 -- 172.21.15.1:0/2884812546 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f00c8018980 con 0x7f00cc108dd0 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f00b80396a0 msgr2=0x7f00b803bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f00b80396a0 0x7f00b803bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f00bc006fd0 tx=0x7f00bc006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 msgr2=0x7f00cc19cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc19cc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00c8000c00 tx=0x7f00c8011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 shutdown_connections 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f00b80396a0 0x7f00b803bb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 --2- 172.21.15.1:0/2884812546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00cc108dd0 0x7f00cc19cc80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 >> 172.21.15.1:0/2884812546 conn(0x7f00cc104370 msgr2=0x7f00cc104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 shutdown_connections 2024-04-26T22:40:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:36.794+0000 7f00d46af700 1 -- 172.21.15.1:0/2884812546 wait complete. 2024-04-26T22:40:38.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.688+0000 7f690ebb9700 1 Processor -- start 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.688+0000 7f690ebb9700 1 -- start start 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690ebb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f69081091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690ebb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f69081097c0 con 0x7f6908108dd0 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690c955700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f69081091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690c955700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f69081091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48914/0 (socket says 172.21.15.1:48914) 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690c955700 1 -- 172.21.15.1:0/847395314 learned_addr learned my addr 172.21.15.1:0/847395314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690c955700 1 -- 172.21.15.1:0/847395314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6908109f80 con 0x7f6908108dd0 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f690c955700 1 --2- 172.21.15.1:0/847395314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f69081091f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f68fc0098a0 tx=0x7f68fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a28080d2bb9192 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f69077fe700 1 -- 172.21.15.1:0/847395314 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68fc004160 con 0x7f6908108dd0 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f69077fe700 1 -- 172.21.15.1:0/847395314 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68fc004300 con 0x7f6908108dd0 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.689+0000 7f69077fe700 1 -- 172.21.15.1:0/847395314 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68fc011a20 con 0x7f6908108dd0 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 -- 172.21.15.1:0/847395314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 msgr2=0x7f69081091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 --2- 172.21.15.1:0/847395314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f69081091f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f68fc0098a0 tx=0x7f68fc009bb0 comp rx=0 tx=0).stop 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 -- 172.21.15.1:0/847395314 shutdown_connections 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 --2- 172.21.15.1:0/847395314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f69081091f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 -- 172.21.15.1:0/847395314 >> 172.21.15.1:0/847395314 conn(0x7f6908104370 msgr2=0x7f69081067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 -- 172.21.15.1:0/847395314 shutdown_connections 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.690+0000 7f690ebb9700 1 -- 172.21.15.1:0/847395314 wait complete. 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690ebb9700 1 Processor -- start 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690ebb9700 1 -- start start 2024-04-26T22:40:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690ebb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f690819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690ebb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f690810d330 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690c955700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f690819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690c955700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f690819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48924/0 (socket says 172.21.15.1:48924) 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690c955700 1 -- 172.21.15.1:0/4235469333 learned_addr learned my addr 172.21.15.1:0/4235469333 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690c955700 1 -- 172.21.15.1:0/4235469333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690819d1e0 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.691+0000 7f690c955700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f690819cca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f68fc000c00 tx=0x7f68fc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.692+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68fc003aa0 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.692+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68fc01b8f0 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.692+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f690819d470 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.692+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68fc023890 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.692+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69081a0160 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.693+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f68fc02c3f0 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.693+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6908064a80 con 0x7f6908108dd0 2024-04-26T22:40:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.693+0000 7f6905ffb700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68f8039700 0x7f68f803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.693+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f68fc04e140 con 0x7f6908108dd0 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.693+0000 7f6907fff700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68f8039700 0x7f68f803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.694+0000 7f6907fff700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68f8039700 0x7f68f803bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f68f4006fd0 tx=0x7f68f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:37.698+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f68fc01f030 con 0x7f6908108dd0 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.021+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f69081a03d0 con 0x7f6908108dd0 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.023+0000 7f6905ffb700 1 -- 172.21.15.1:0/4235469333 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f68fc01b440 con 0x7f6908108dd0 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68f8039700 msgr2=0x7f68f803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68f8039700 0x7f68f803bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f68f4006fd0 tx=0x7f68f4006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 msgr2=0x7f690819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f690819cca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f68fc000c00 tx=0x7f68fc011aa0 comp rx=0 tx=0).stop 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 shutdown_connections 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68f8039700 0x7f68f803bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 --2- 172.21.15.1:0/4235469333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6908108dd0 0x7f690819cca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 >> 172.21.15.1:0/4235469333 conn(0x7f6908104370 msgr2=0x7f6908104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 shutdown_connections 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T22:40:38.025+0000 7f690ebb9700 1 -- 172.21.15.1:0/4235469333 wait complete. 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T22:40:38.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout: ceph telemetry on 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout:For more information see: 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:38.636 INFO:teuthology.orchestra.run.smithi001.stdout:Bootstrap complete. 2024-04-26T22:40:38.675 INFO:tasks.cephadm:Fetching config... 2024-04-26T22:40:38.676 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:40:38.676 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T22:40:38.693 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T22:40:38.693 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:40:38.693 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T22:40:38.749 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T22:40:38.749 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:40:38.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T22:40:38.815 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T22:40:38.815 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:40:38.815 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T22:40:38.871 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T22:40:38.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T22:40:38.954 INFO:teuthology.orchestra.run.smithi001.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:40:38.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T22:40:39.013 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:40:39.026 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T22:40:39.068 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0hhmsApW1iDhR96AF+KNS7Q6YkUARu8Ce3JXqBMHUY5MXJ/rIBp0kEy7XkUORIGrxUG5EaihS9U+BcTSwzZkPhe6n4hNcHVI/Uzx/ick4wGz7vZpR10NCVRcfTgQw6TCZFgtyQRrSHBi+TFJVgJSHm4giFahVTMKfjFUBZ5/osYo6fDol+J+FyieqK7gfXdCYcFwpXjsGiNlPoMUPi1mPIV92cH50WN0Y1BPwK4R/DDMNJjF1YRDAIXFuENvoPRiKXok/IiLESzk/X0hxCJeUo3pwRN0yHQF3ELPiD3LlPmHRXDHkhHF0EUNI9TwJLEf7dT3qUiCa5gIJ/u5NLvJVRhm73zOltywstlMxbqwulS9YeXW+exG2T585DeGN12NA7EPBUOVJPpdlTqdJYB8G6sSWIVljdJKHKm9vfWbOuWB3lRBhUSbMQOad9qb02W5jbTVuwH+qtuqFhSRpM9R09jMrRivg9l6Ckz8JasrC+VBAF6rZ+MgoNbopsrIHKN0= ceph-bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:40:39.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T22:40:39.201 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:39 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4235469333' entity='client.admin' 2024-04-26T22:40:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.543+0000 7f42726e4700 1 -- 172.21.15.1:0/1639270442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 msgr2=0x7f426c103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.543+0000 7f42726e4700 1 --2- 172.21.15.1:0/1639270442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c103270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f425c009910 tx=0x7f425c009c20 comp rx=0 tx=0).stop 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.543+0000 7f42726e4700 1 -- 172.21.15.1:0/1639270442 shutdown_connections 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.543+0000 7f42726e4700 1 --2- 172.21.15.1:0/1639270442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c103270 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.543+0000 7f42726e4700 1 -- 172.21.15.1:0/1639270442 >> 172.21.15.1:0/1639270442 conn(0x7f426c0fe440 msgr2=0x7f426c100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.544+0000 7f42726e4700 1 -- 172.21.15.1:0/1639270442 shutdown_connections 2024-04-26T22:40:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.544+0000 7f42726e4700 1 -- 172.21.15.1:0/1639270442 wait complete. 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.544+0000 7f42726e4700 1 Processor -- start 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.544+0000 7f42726e4700 1 -- start start 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.544+0000 7f42726e4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.544+0000 7f42726e4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f426c108d50 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f426bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f426bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48946/0 (socket says 172.21.15.1:48946) 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f426bfff700 1 -- 172.21.15.1:0/2487053855 learned_addr learned my addr 172.21.15.1:0/2487053855 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f426bfff700 1 -- 172.21.15.1:0/2487053855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426c198a20 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f426bfff700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c1984e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f425c005800 tx=0x7f425c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f425c027070 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f425c00be10 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f426c198cb0 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f425c00f460 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.545+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426c19b9a0 con 0x7f426c102e50 2024-04-26T22:40:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.546+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f425c02a070 con 0x7f426c102e50 2024-04-26T22:40:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.546+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f426c1094d0 con 0x7f426c102e50 2024-04-26T22:40:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.546+0000 7f42697fa700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4254039650 0x7f425403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.546+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f425c01e030 con 0x7f426c102e50 2024-04-26T22:40:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.546+0000 7f426b7fe700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4254039650 0x7f425403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.547+0000 7f426b7fe700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4254039650 0x7f425403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4260006fd0 tx=0x7f4260006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.552+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f425c013070 con 0x7f426c102e50 2024-04-26T22:40:41.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.762+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f426c02e5a0 con 0x7f426c102e50 2024-04-26T22:40:41.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.765+0000 7f42697fa700 1 -- 172.21.15.1:0/2487053855 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f425c031020 con 0x7f426c102e50 2024-04-26T22:40:41.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.767+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4254039650 msgr2=0x7f425403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:41.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.767+0000 7f42726e4700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4254039650 0x7f425403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4260006fd0 tx=0x7f4260006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:41.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.767+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 msgr2=0x7f426c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:41.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.767+0000 7f42726e4700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c1984e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f425c005800 tx=0x7f425c009f60 comp rx=0 tx=0).stop 2024-04-26T22:40:41.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.767+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 shutdown_connections 2024-04-26T22:40:41.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.768+0000 7f42726e4700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4254039650 0x7f425403bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:41.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.768+0000 7f42726e4700 1 --2- 172.21.15.1:0/2487053855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f426c102e50 0x7f426c1984e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:41.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.768+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 >> 172.21.15.1:0/2487053855 conn(0x7f426c0fe440 msgr2=0x7f426c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:41.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.768+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 shutdown_connections 2024-04-26T22:40:41.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:41.768+0000 7f42726e4700 1 -- 172.21.15.1:0/2487053855 wait complete. 2024-04-26T22:40:42.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:42.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:42.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:40:42.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:42.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:42.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:42.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2487053855' entity='client.admin' 2024-04-26T22:40:42.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:42.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:42.379 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T22:40:42.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T22:40:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:40:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 -- 172.21.15.1:0/1785509012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 msgr2=0x7f5264103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 --2- 172.21.15.1:0/1785509012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f5264103290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5254009910 tx=0x7f5254009c20 comp rx=0 tx=0).stop 2024-04-26T22:40:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 -- 172.21.15.1:0/1785509012 shutdown_connections 2024-04-26T22:40:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 --2- 172.21.15.1:0/1785509012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f5264103290 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 -- 172.21.15.1:0/1785509012 >> 172.21.15.1:0/1785509012 conn(0x7f52640fe420 msgr2=0x7f52641008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 -- 172.21.15.1:0/1785509012 shutdown_connections 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.108+0000 7f526b2a9700 1 -- 172.21.15.1:0/1785509012 wait complete. 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f526b2a9700 1 Processor -- start 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f526b2a9700 1 -- start start 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f526b2a9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f52640770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f526b2a9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5264108d70 con 0x7f5264102e70 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f5269045700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f52640770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f5269045700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f52640770e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46554/0 (socket says 172.21.15.1:46554) 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.109+0000 7f5269045700 1 -- 172.21.15.1:0/1681876566 learned_addr learned my addr 172.21.15.1:0/1681876566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f5269045700 1 -- 172.21.15.1:0/1681876566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5264077620 con 0x7f5264102e70 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f5269045700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f52640770e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5254005800 tx=0x7f5254009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5254027070 con 0x7f5264102e70 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f525400be10 con 0x7f5264102e70 2024-04-26T22:40:44.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5264075810 con 0x7f5264102e70 2024-04-26T22:40:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f525400f460 con 0x7f5264102e70 2024-04-26T22:40:44.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.110+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5264075d50 con 0x7f5264102e70 2024-04-26T22:40:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.111+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f525402a070 con 0x7f5264102e70 2024-04-26T22:40:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.111+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5248005360 con 0x7f5264102e70 2024-04-26T22:40:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.111+0000 7f525a7fc700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f52500396b0 0x7f525003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.111+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f525401e030 con 0x7f5264102e70 2024-04-26T22:40:44.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.111+0000 7f5268844700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f52500396b0 0x7f525003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:44.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.112+0000 7f5268844700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f52500396b0 0x7f525003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5260006fd0 tx=0x7f5260006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:44.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.116+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5254013070 con 0x7f5264102e70 2024-04-26T22:40:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.333+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f5248000bf0 con 0x7f52500396b0 2024-04-26T22:40:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.336+0000 7f525a7fc700 1 -- 172.21.15.1:0/1681876566 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5248000bf0 con 0x7f52500396b0 2024-04-26T22:40:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.338+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f52500396b0 msgr2=0x7f525003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.338+0000 7f526b2a9700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f52500396b0 0x7f525003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5260006fd0 tx=0x7f5260006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.338+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 msgr2=0x7f52640770e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.338+0000 7f526b2a9700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f52640770e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5254005800 tx=0x7f5254009f60 comp rx=0 tx=0).stop 2024-04-26T22:40:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.339+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 shutdown_connections 2024-04-26T22:40:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.339+0000 7f526b2a9700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f52500396b0 0x7f525003bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.339+0000 7f526b2a9700 1 --2- 172.21.15.1:0/1681876566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5264102e70 0x7f52640770e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.339+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 >> 172.21.15.1:0/1681876566 conn(0x7f52640fe420 msgr2=0x7f52640fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.339+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 shutdown_connections 2024-04-26T22:40:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:44.339+0000 7f526b2a9700 1 -- 172.21.15.1:0/1681876566 wait complete. 2024-04-26T22:40:45.131 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi038 2024-04-26T22:40:45.131 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:40:45.132 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.conf 2024-04-26T22:40:45.151 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:40:45.151 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:40:45.211 INFO:tasks.cephadm:Adding host smithi038 to orchestrator... 2024-04-26T22:40:45.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch host add smithi038 2024-04-26T22:40:45.453 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:40:45.575 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:45.575 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:45.575 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:45.575 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:40:45.575 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:40:45.576 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T22:40:45.576 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:45.576 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:45.576 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:46.425 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:46 smithi001 ceph-mon[29435]: Updating smithi001:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:40:46.425 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:46 smithi001 ceph-mon[29435]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:40:46.425 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:46 smithi001 ceph-mon[29435]: Updating smithi001:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T22:40:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 -- 172.21.15.1:0/1263283323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 msgr2=0x7fa3d8103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 --2- 172.21.15.1:0/1263283323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d8103470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009910 tx=0x7fa3c0009c20 comp rx=0 tx=0).stop 2024-04-26T22:40:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 -- 172.21.15.1:0/1263283323 shutdown_connections 2024-04-26T22:40:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 --2- 172.21.15.1:0/1263283323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d8103470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 -- 172.21.15.1:0/1263283323 >> 172.21.15.1:0/1263283323 conn(0x7fa3d80fe960 msgr2=0x7fa3d8100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 -- 172.21.15.1:0/1263283323 shutdown_connections 2024-04-26T22:40:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.953+0000 7fa3de376700 1 -- 172.21.15.1:0/1263283323 wait complete. 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.954+0000 7fa3de376700 1 Processor -- start 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.954+0000 7fa3de376700 1 -- start start 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.954+0000 7fa3de376700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.954+0000 7fa3de376700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d810d310 con 0x7fa3d8103090 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.954+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d819e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46566/0 (socket says 172.21.15.1:46566) 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d7fff700 1 -- 172.21.15.1:0/1547375194 learned_addr learned my addr 172.21.15.1:0/1547375194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d7fff700 1 -- 172.21.15.1:0/1547375194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d819eaf0 con 0x7fa3d8103090 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d7fff700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d819e5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00056d0 tx=0x7fa3c00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3c0026070 con 0x7fa3d8103090 2024-04-26T22:40:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c000bba0 con 0x7fa3d8103090 2024-04-26T22:40:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d819ede0 con 0x7fa3d8103090 2024-04-26T22:40:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3c0003920 con 0x7fa3d8103090 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.955+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d8198850 con 0x7fa3d8103090 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.956+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fa3c0029070 con 0x7fa3d8103090 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.956+0000 7fa3d57fa700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa3c40396b0 0x7fa3c403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.956+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa3c001e070 con 0x7fa3d8103090 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.956+0000 7fa3d77fe700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa3c40396b0 0x7fa3c403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.956+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d8104570 con 0x7fa3d8103090 2024-04-26T22:40:46.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.957+0000 7fa3d77fe700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa3c40396b0 0x7fa3c403bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8006fd0 tx=0x7fa3c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:46.962+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa3c0013070 con 0x7fa3d8103090 2024-04-26T22:40:47.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:47.175+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}) -- 0x7fa3d819efb0 con 0x7fa3c40396b0 2024-04-26T22:40:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:47 smithi001 ceph-mon[29435]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:49.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:48 smithi001 ceph-mon[29435]: Deploying cephadm binary to smithi038 2024-04-26T22:40:51.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:50 smithi001 ceph-mon[29435]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:40:51.447 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi038' with addr '172.21.15.38' 2024-04-26T22:40:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.446+0000 7fa3d57fa700 1 -- 172.21.15.1:0/1547375194 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fa3d819efb0 con 0x7fa3c40396b0 2024-04-26T22:40:51.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa3c40396b0 msgr2=0x7fa3c403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa3c40396b0 0x7fa3c403bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8006fd0 tx=0x7fa3c8006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 msgr2=0x7fa3d819e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d819e5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00056d0 tx=0x7fa3c00057b0 comp rx=0 tx=0).stop 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 shutdown_connections 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa3c40396b0 0x7fa3c403bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 --2- 172.21.15.1:0/1547375194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d8103090 0x7fa3d819e5b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 >> 172.21.15.1:0/1547375194 conn(0x7fa3d80fe960 msgr2=0x7fa3d8104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 shutdown_connections 2024-04-26T22:40:51.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:51.449+0000 7fa3de376700 1 -- 172.21.15.1:0/1547375194 wait complete. 2024-04-26T22:40:52.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T22:40:52.485 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:40:52.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:52.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:52 smithi001 ceph-mon[29435]: Added host smithi038 2024-04-26T22:40:52.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:52.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:53.638 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:53 smithi001 ceph-mon[29435]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:40:53.638 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:53 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 -- 172.21.15.1:0/3665206947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 msgr2=0x7fbe88109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 --2- 172.21.15.1:0/3665206947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe88109390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000b0a0 tx=0x7fbe8000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 -- 172.21.15.1:0/3665206947 shutdown_connections 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 --2- 172.21.15.1:0/3665206947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe88109390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 -- 172.21.15.1:0/3665206947 >> 172.21.15.1:0/3665206947 conn(0x7fbe880784a0 msgr2=0x7fbe880788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 -- 172.21.15.1:0/3665206947 shutdown_connections 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.130+0000 7fbe8f18a700 1 -- 172.21.15.1:0/3665206947 wait complete. 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8f18a700 1 Processor -- start 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8f18a700 1 -- start start 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8f18a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe8819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8f18a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8810d460 con 0x7fbe88108fb0 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8cf26700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe8819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8cf26700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe8819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60648/0 (socket says 172.21.15.1:60648) 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.131+0000 7fbe8cf26700 1 -- 172.21.15.1:0/2264664315 learned_addr learned my addr 172.21.15.1:0/2264664315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe8cf26700 1 -- 172.21.15.1:0/2264664315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8819ebc0 con 0x7fbe88108fb0 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe8cf26700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe8819e680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000d010 tx=0x7fbe80003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe80021020 con 0x7fbe88108fb0 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe800096a0 con 0x7fbe88108fb0 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8819ee50 con 0x7fbe88108fb0 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe80017040 con 0x7fbe88108fb0 2024-04-26T22:40:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.132+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe88198750 con 0x7fbe88108fb0 2024-04-26T22:40:54.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.133+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fbe80024020 con 0x7fbe88108fb0 2024-04-26T22:40:54.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.133+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe881046d0 con 0x7fbe88108fb0 2024-04-26T22:40:54.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.133+0000 7fbe85ffb700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbe740396a0 0x7fbe7403bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:54.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.133+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe8000f050 con 0x7fbe88108fb0 2024-04-26T22:40:54.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.133+0000 7fbe87fff700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbe740396a0 0x7fbe7403bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:54.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.134+0000 7fbe87fff700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbe740396a0 0x7fbe7403bb60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe78006fd0 tx=0x7fbe78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.138+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe8002b020 con 0x7fbe88108fb0 2024-04-26T22:40:54.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.354+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe88199c80 con 0x7fbe740396a0 2024-04-26T22:40:54.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.355+0000 7fbe85ffb700 1 -- 172.21.15.1:0/2264664315 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+158 (secure 0 0 0) 0x7fbe88199c80 con 0x7fbe740396a0 2024-04-26T22:40:54.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:40:54.356 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}] 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbe740396a0 msgr2=0x7fbe7403bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbe740396a0 0x7fbe7403bb60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe78006fd0 tx=0x7fbe78006e40 comp rx=0 tx=0).stop 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 msgr2=0x7fbe8819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe8819e680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000d010 tx=0x7fbe80003ce0 comp rx=0 tx=0).stop 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 shutdown_connections 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbe740396a0 0x7fbe7403bb60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 --2- 172.21.15.1:0/2264664315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe88108fb0 0x7fbe8819e680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 >> 172.21.15.1:0/2264664315 conn(0x7fbe880784a0 msgr2=0x7fbe88104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:54.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 shutdown_connections 2024-04-26T22:40:54.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:54.357+0000 7fbe8f18a700 1 -- 172.21.15.1:0/2264664315 wait complete. 2024-04-26T22:40:54.971 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-04-26T22:40:54.972 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:40:54.972 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-04-26T22:40:54.989 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:40:54.989 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:40:55.046 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-04-26T22:40:55.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch host add smithi186 2024-04-26T22:40:55.286 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:40:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:55 smithi001 ceph-mon[29435]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:40:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:55 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:55 smithi001 ceph-mon[29435]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:40:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 -- 172.21.15.1:0/2717244391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 msgr2=0x7f10dc1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:40:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 --2- 172.21.15.1:0/2717244391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1023b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00b0a0 tx=0x7f10cc00b3b0 comp rx=0 tx=0).stop 2024-04-26T22:40:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 -- 172.21.15.1:0/2717244391 shutdown_connections 2024-04-26T22:40:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 --2- 172.21.15.1:0/2717244391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1023b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 -- 172.21.15.1:0/2717244391 >> 172.21.15.1:0/2717244391 conn(0x7f10dc0fd880 msgr2=0x7f10dc0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 -- 172.21.15.1:0/2717244391 shutdown_connections 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 -- 172.21.15.1:0/2717244391 wait complete. 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.742+0000 7f10e1aae700 1 Processor -- start 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10e1aae700 1 -- start start 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10e1aae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10e1aae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc10c250 con 0x7f10dc101fd0 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10db7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10db7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60666/0 (socket says 172.21.15.1:60666) 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10db7fe700 1 -- 172.21.15.1:0/547532292 learned_addr learned my addr 172.21.15.1:0/547532292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10db7fe700 1 -- 172.21.15.1:0/547532292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10dc1a0c10 con 0x7f10dc101fd0 2024-04-26T22:40:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10db7fe700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1a06d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f10cc006130 tx=0x7f10cc0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10cc021020 con 0x7f10dc101fd0 2024-04-26T22:40:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10cc0079b0 con 0x7f10dc101fd0 2024-04-26T22:40:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10dc1a0ea0 con 0x7f10dc101fd0 2024-04-26T22:40:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.743+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10cc017040 con 0x7f10dc101fd0 2024-04-26T22:40:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.744+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10dc19a890 con 0x7f10dc101fd0 2024-04-26T22:40:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.745+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f10cc024020 con 0x7f10dc101fd0 2024-04-26T22:40:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.745+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10dc1034b0 con 0x7f10dc101fd0 2024-04-26T22:40:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.745+0000 7f10d8ff9700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10c8039650 0x7f10c803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:40:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.745+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f10cc00f050 con 0x7f10dc101fd0 2024-04-26T22:40:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.745+0000 7f10daffd700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10c8039650 0x7f10c803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:40:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.745+0000 7f10daffd700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10c8039650 0x7f10c803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10c4006fd0 tx=0x7f10c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:40:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.750+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f10cc02b020 con 0x7f10dc101fd0 2024-04-26T22:40:56.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:40:56.961+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f10dc19bdc0 con 0x7f10c8039650 2024-04-26T22:40:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:57 smithi001 ceph-mon[29435]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:40:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:40:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:40:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:40:58.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:40:58.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:40:58.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T22:40:58.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:40:58.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:58 smithi001 ceph-mon[29435]: Deploying cephadm binary to smithi186 2024-04-26T22:40:59.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:59 smithi001 ceph-mon[29435]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:40:59.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:59 smithi001 ceph-mon[29435]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:40:59.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:40:59 smithi001 ceph-mon[29435]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T22:41:01.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.163+0000 7f10d8ff9700 1 -- 172.21.15.1:0/547532292 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f10dc19bdc0 con 0x7f10c8039650 2024-04-26T22:41:01.164 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10c8039650 msgr2=0x7f10c803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10c8039650 0x7f10c803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10c4006fd0 tx=0x7f10c4006e40 comp rx=0 tx=0).stop 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 msgr2=0x7f10dc1a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1a06d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f10cc006130 tx=0x7f10cc0097d0 comp rx=0 tx=0).stop 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 shutdown_connections 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10c8039650 0x7f10c803bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 --2- 172.21.15.1:0/547532292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc101fd0 0x7f10dc1a06d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:01.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.165+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 >> 172.21.15.1:0/547532292 conn(0x7f10dc0fd880 msgr2=0x7f10dc06b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:01.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.166+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 shutdown_connections 2024-04-26T22:41:01.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:01.166+0000 7f10e1aae700 1 -- 172.21.15.1:0/547532292 wait complete. 2024-04-26T22:41:01.499 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:01 smithi001 ceph-mon[29435]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:01.499 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:01.499 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:01.499 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:01.499 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:01.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T22:41:02.077 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:41:02.502 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:02 smithi001 ceph-mon[29435]: Added host smithi186 2024-04-26T22:41:02.502 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:02.502 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:03.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:03 smithi001 ceph-mon[29435]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:03.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 -- 172.21.15.1:0/308897691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 msgr2=0x7f7ed81034a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 --2- 172.21.15.1:0/308897691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81034a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009910 tx=0x7f7ec8009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 -- 172.21.15.1:0/308897691 shutdown_connections 2024-04-26T22:41:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 --2- 172.21.15.1:0/308897691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81034a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 -- 172.21.15.1:0/308897691 >> 172.21.15.1:0/308897691 conn(0x7f7ed80fe950 msgr2=0x7f7ed8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 -- 172.21.15.1:0/308897691 shutdown_connections 2024-04-26T22:41:03.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.656+0000 7f7eddf9e700 1 -- 172.21.15.1:0/308897691 wait complete. 2024-04-26T22:41:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7eddf9e700 1 Processor -- start 2024-04-26T22:41:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7eddf9e700 1 -- start start 2024-04-26T22:41:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7eddf9e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:03.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7eddf9e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810d340 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7ed77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7ed77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58376/0 (socket says 172.21.15.1:58376) 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.657+0000 7f7ed77fe700 1 -- 172.21.15.1:0/1240711895 learned_addr learned my addr 172.21.15.1:0/1240711895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7ed77fe700 1 -- 172.21.15.1:0/1240711895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed81a0c30 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7ed77fe700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81a06f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80056d0 tx=0x7f7ec80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ec8026070 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec800bba0 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed81a0f20 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ec8003920 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.658+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed819a8b0 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.659+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f7ec8029070 con 0x7f7ed81030c0 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.659+0000 7f7ed4ff9700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7ec00396b0 0x7f7ec003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:03.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.659+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7ec801e070 con 0x7f7ed81030c0 2024-04-26T22:41:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.659+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed81045a0 con 0x7f7ed81030c0 2024-04-26T22:41:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.659+0000 7f7ed6ffd700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7ec00396b0 0x7f7ec003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:03.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.660+0000 7f7ed6ffd700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7ec00396b0 0x7f7ec003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc006fd0 tx=0x7f7ecc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:03.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.664+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7ec8030070 con 0x7f7ed81030c0 2024-04-26T22:41:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.895+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7ed819bef0 con 0x7f7ec00396b0 2024-04-26T22:41:03.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.896+0000 7f7ed4ff9700 1 -- 172.21.15.1:0/1240711895 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7f7ed819bef0 con 0x7f7ec00396b0 2024-04-26T22:41:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:41:03.897 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-04-26T22:41:03.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7ec00396b0 msgr2=0x7f7ec003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:03.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7ec00396b0 0x7f7ec003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc006fd0 tx=0x7f7ecc006e40 comp rx=0 tx=0).stop 2024-04-26T22:41:03.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 msgr2=0x7f7ed81a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81a06f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80056d0 tx=0x7f7ec80057b0 comp rx=0 tx=0).stop 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 shutdown_connections 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7ec00396b0 0x7f7ec003bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 --2- 172.21.15.1:0/1240711895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ed81030c0 0x7f7ed81a06f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 >> 172.21.15.1:0/1240711895 conn(0x7f7ed80fe950 msgr2=0x7f7ed8075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 shutdown_connections 2024-04-26T22:41:03.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:03.899+0000 7f7eddf9e700 1 -- 172.21.15.1:0/1240711895 wait complete. 2024-04-26T22:41:04.481 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T22:41:04.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T22:41:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:41:05.064 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:04 smithi001 ceph-mon[29435]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:05.064 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:05.064 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:04 smithi001 ceph-mon[29435]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:41:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.220+0000 7f944c047700 1 -- 172.21.15.1:0/610930621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 msgr2=0x7f9444104410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.220+0000 7f944c047700 1 --2- 172.21.15.1:0/610930621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444104410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9434009910 tx=0x7f9434009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 -- 172.21.15.1:0/610930621 shutdown_connections 2024-04-26T22:41:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 --2- 172.21.15.1:0/610930621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444104410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 -- 172.21.15.1:0/610930621 >> 172.21.15.1:0/610930621 conn(0x7f94440fd880 msgr2=0x7f94440ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 -- 172.21.15.1:0/610930621 shutdown_connections 2024-04-26T22:41:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 -- 172.21.15.1:0/610930621 wait complete. 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 Processor -- start 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 -- start start 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.221+0000 7f944c047700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f944410d3b0 con 0x7f9444104030 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f9449de3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f9449de3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58386/0 (socket says 172.21.15.1:58386) 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f9449de3700 1 -- 172.21.15.1:0/2546668218 learned_addr learned my addr 172.21.15.1:0/2546668218 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f9449de3700 1 -- 172.21.15.1:0/2546668218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9444103b80 con 0x7f9444104030 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f9449de3700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444101c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f94340056d0 tx=0x7f94340057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9434026070 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94441021d0 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9434004370 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94340039b0 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.222+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9444102690 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.223+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f9434029070 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.223+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f944406bb70 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.223+0000 7f943affd700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9430039650 0x7f943003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.223+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f943401e070 con 0x7f9444104030 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.224+0000 7f94495e2700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9430039650 0x7f943003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.224+0000 7f94495e2700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9430039650 0x7f943003bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9440006fd0 tx=0x7f9440006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.228+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9434013070 con 0x7f9444104030 2024-04-26T22:41:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.442+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f94441031c0 con 0x7f9444104030 2024-04-26T22:41:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.773+0000 7f943affd700 1 -- 172.21.15.1:0/2546668218 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f943400d340 con 0x7f9444104030 2024-04-26T22:41:06.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9430039650 msgr2=0x7f943003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:06.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9430039650 0x7f943003bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9440006fd0 tx=0x7f9440006e40 comp rx=0 tx=0).stop 2024-04-26T22:41:06.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 msgr2=0x7f9444101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444101c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f94340056d0 tx=0x7f94340057b0 comp rx=0 tx=0).stop 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 shutdown_connections 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9430039650 0x7f943003bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 --2- 172.21.15.1:0/2546668218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9444104030 0x7f9444101c90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 >> 172.21.15.1:0/2546668218 conn(0x7f94440fd880 msgr2=0x7f9444104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 shutdown_connections 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:06.776+0000 7f944c047700 1 -- 172.21.15.1:0/2546668218 wait complete. 2024-04-26T22:41:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-04-26T22:41:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:06 smithi001 ceph-mon[29435]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:06 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2546668218' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T22:41:07.446 INFO:tasks.cephadm:Adding mon.a on smithi001 2024-04-26T22:41:07.446 INFO:tasks.cephadm:Adding mon.b on smithi038 2024-04-26T22:41:07.446 INFO:tasks.cephadm:Adding mon.c on smithi186 2024-04-26T22:41:07.447 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c' 2024-04-26T22:41:07.679 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2546668218' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T22:41:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:09.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:08 smithi001 ceph-mon[29435]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:09.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 -- 172.21.15.186:0/1466839772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 msgr2=0x7f1124103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:09.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 --2- 172.21.15.186:0/1466839772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f1124103470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1120009910 tx=0x7f1120009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:09.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 -- 172.21.15.186:0/1466839772 shutdown_connections 2024-04-26T22:41:09.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 --2- 172.21.15.186:0/1466839772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f1124103470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:09.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 -- 172.21.15.186:0/1466839772 >> 172.21.15.186:0/1466839772 conn(0x7f11240fe960 msgr2=0x7f1124100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 -- 172.21.15.186:0/1466839772 shutdown_connections 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.224+0000 7f112c29f700 1 -- 172.21.15.186:0/1466839772 wait complete. 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112c29f700 1 Processor -- start 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112c29f700 1 -- start start 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112c29f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f112419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112c29f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f112410d310 con 0x7f1124103090 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112a03b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f112419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:09.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112a03b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f112419e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.186:45724/0 (socket says 172.21.15.186:45724) 2024-04-26T22:41:09.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.225+0000 7f112a03b700 1 -- 172.21.15.186:0/470063844 learned_addr learned my addr 172.21.15.186:0/470063844 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:41:09.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f112a03b700 1 -- 172.21.15.186:0/470063844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112419eb10 con 0x7f1124103090 2024-04-26T22:41:09.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f112a03b700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f112419e5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f11200056d0 tx=0x7f11200057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:09.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1120026070 con 0x7f1124103090 2024-04-26T22:41:09.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112419ee00 con 0x7f1124103090 2024-04-26T22:41:09.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f112000bba0 con 0x7f1124103090 2024-04-26T22:41:09.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1124198870 con 0x7f1124103090 2024-04-26T22:41:09.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.226+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1120003910 con 0x7f1124103090 2024-04-26T22:41:09.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.227+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1124104570 con 0x7f1124103090 2024-04-26T22:41:09.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.227+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f1120029070 con 0x7f1124103090 2024-04-26T22:41:09.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.227+0000 7f111b7fe700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1110039650 0x7f111003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:09.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.228+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f112001e070 con 0x7f1124103090 2024-04-26T22:41:09.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.228+0000 7f112983a700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1110039650 0x7f111003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:09.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.228+0000 7f112983a700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1110039650 0x7f111003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1114006fd0 tx=0x7f1114006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:09.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.234+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1120013070 con 0x7f1124103090 2024-04-26T22:41:09.442 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.441+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}) -- 0x7f11240008d0 con 0x7f1110039650 2024-04-26T22:41:09.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.448+0000 7f111b7fe700 1 -- 172.21.15.186:0/470063844 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f11240008d0 con 0x7f1110039650 2024-04-26T22:41:09.449 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-04-26T22:41:09.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1110039650 msgr2=0x7f111003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:09.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1110039650 0x7f111003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1114006fd0 tx=0x7f1114006e40 comp rx=0 tx=0).stop 2024-04-26T22:41:09.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 msgr2=0x7f112419e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:09.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f112419e5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f11200056d0 tx=0x7f11200057b0 comp rx=0 tx=0).stop 2024-04-26T22:41:09.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 shutdown_connections 2024-04-26T22:41:09.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1110039650 0x7f111003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:09.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 --2- 172.21.15.186:0/470063844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1124103090 0x7f112419e5d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:09.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 >> 172.21.15.186:0/470063844 conn(0x7f11240fe960 msgr2=0x7f1124104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:09.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 shutdown_connections 2024-04-26T22:41:09.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:09.450+0000 7f112c29f700 1 -- 172.21.15.186:0/470063844 wait complete. 2024-04-26T22:41:10.079 DEBUG:teuthology.orchestra.run.smithi038:mon.b> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T22:41:10.081 DEBUG:teuthology.orchestra.run.smithi186:mon.c> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T22:41:10.084 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T22:41:10.084 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T22:41:10.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:10.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: Saving service mon spec with placement smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c;count:3 2024-04-26T22:41:10.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:10.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:10 smithi001 ceph-mon[29435]: Deploying daemon mon.c on smithi186 2024-04-26T22:41:11.040 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:11 smithi001 ceph-mon[29435]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:13.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 -- 172.21.15.186:0/573216351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 msgr2=0x7f0c001043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:13.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 --2- 172.21.15.186:0/573216351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001043d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009910 tx=0x7f0bf0009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 -- 172.21.15.186:0/573216351 shutdown_connections 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 --2- 172.21.15.186:0/573216351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001043d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 -- 172.21.15.186:0/573216351 >> 172.21.15.186:0/573216351 conn(0x7f0c000fd840 msgr2=0x7f0c000ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 -- 172.21.15.186:0/573216351 shutdown_connections 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 -- 172.21.15.186:0/573216351 wait complete. 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.175+0000 7f0c07512700 1 Processor -- start 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c07512700 1 -- start start 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c07512700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:13.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c07512700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0010d390 con 0x7f0c00103ff0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c052ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c052ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.186:45742/0 (socket says 172.21.15.186:45742) 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c052ae700 1 -- 172.21.15.186:0/61768149 learned_addr learned my addr 172.21.15.186:0/61768149 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.176+0000 7f0c052ae700 1 -- 172.21.15.186:0/61768149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c001a0cc0 con 0x7f0c00103ff0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.177+0000 7f0c052ae700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001a0780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00056d0 tx=0x7f0bf00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.177+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf0026070 con 0x7f0c00103ff0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.177+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c001a0f50 con 0x7f0c00103ff0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.177+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bf0004370 con 0x7f0c00103ff0 2024-04-26T22:41:13.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.177+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf00039b0 con 0x7f0c00103ff0 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.177+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0019a990 con 0x7f0c00103ff0 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.178+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0006bbb0 con 0x7f0c00103ff0 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.178+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f0bf0029070 con 0x7f0c00103ff0 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.178+0000 7f0bf67fc700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0bec039650 0x7f0bec03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.178+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0bf001e070 con 0x7f0c00103ff0 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.178+0000 7f0c04aad700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0bec039650 0x7f0bec03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:13.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.179+0000 7f0c04aad700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0bec039650 0x7f0bec03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc006fd0 tx=0x7f0bfc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:13.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.183+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0bf0013070 con 0x7f0c00103ff0 2024-04-26T22:41:13.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.468+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0c0019be70 con 0x7f0c00103ff0 2024-04-26T22:41:13.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.469+0000 7f0bf67fc700 1 -- 172.21.15.186:0/61768149 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+741 (secure 0 0 0) 0x7f0bf0030090 con 0x7f0c00103ff0 2024-04-26T22:41:13.468 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:41:13.469 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","modified":"2024-04-26T22:39:23.873268Z","created":"2024-04-26T22:39:23.873268Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T22:41:13.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0bec039650 msgr2=0x7f0bec03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0bec039650 0x7f0bec03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc006fd0 tx=0x7f0bfc006e40 comp rx=0 tx=0).stop 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 msgr2=0x7f0c001a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001a0780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00056d0 tx=0x7f0bf00057b0 comp rx=0 tx=0).stop 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 shutdown_connections 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0bec039650 0x7f0bec03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 --2- 172.21.15.186:0/61768149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c00103ff0 0x7f0c001a0780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 >> 172.21.15.186:0/61768149 conn(0x7f0c000fd840 msgr2=0x7f0c001091b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 shutdown_connections 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:13.471+0000 7f0c07512700 1 -- 172.21.15.186:0/61768149 wait complete. 2024-04-26T22:41:13.471 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-04-26T22:41:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:13 smithi001 ceph-mon[29435]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:14.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:14 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/61768149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:15.129 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T22:41:15.130 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T22:41:15.377 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:41:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:15 smithi186 systemd[1]: Started Ceph mon.c for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:41:20.549 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.546+0000 7f2899c57700 1 -- 172.21.15.186:0/1694974889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f287c0035c0 msgr2=0x7f287c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:20.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.546+0000 7f2899c57700 1 --2- 172.21.15.186:0/1694974889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f287c0035c0 0x7f287c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2894102e70 tx=0x7f2884014ec0 comp rx=0 tx=0).stop 2024-04-26T22:41:20.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.546+0000 7f2899c57700 1 -- 172.21.15.186:0/1694974889 shutdown_connections 2024-04-26T22:41:20.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.546+0000 7f2899c57700 1 --2- 172.21.15.186:0/1694974889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f287c0035c0 0x7f287c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:20.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.546+0000 7f2899c57700 1 -- 172.21.15.186:0/1694974889 >> 172.21.15.186:0/1694974889 conn(0x7f28940fea30 msgr2=0x7f2894100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:20.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.546+0000 7f2899c57700 1 -- 172.21.15.186:0/1694974889 shutdown_connections 2024-04-26T22:41:20.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.547+0000 7f2899c57700 1 -- 172.21.15.186:0/1694974889 wait complete. 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: mon.a calling monitor election 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: mon.c calling monitor election 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T22:41:20.551 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: monmap epoch 2 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: last_changed 2024-04-26T22:41:15.214304+0000 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: min_mon_release 19 (squid) 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: election_strategy: 1 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: fsmap 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: mgrmap e13: x(active, since 50s) 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:20.552 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:20.553 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f2899c57700 1 Processor -- start 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f2899c57700 1 -- start start 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f2899c57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f2899c57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2894075c60 0x7f28940760e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f2899c57700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f289410ae90 con 0x7f2894075c60 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f2899c57700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f289410ad10 con 0x7f287c0035c0 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f28937fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.548+0000 7f28937fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:55288/0 (socket says 172.21.15.186:55288) 2024-04-26T22:41:20.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f28937fe700 1 -- 172.21.15.186:0/786317266 learned_addr learned my addr 172.21.15.186:0/786317266 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f2892ffd700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2894075c60 0x7f28940760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f28937fe700 1 -- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 msgr2=0x7f2894075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f28937fe700 1 -- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 msgr2=0x7f2894075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f28937fe700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f28937fe700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f2892ffd700 1 -- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 msgr2=0x7f2894075720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f2892ffd700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f2892ffd700 1 -- 172.21.15.186:0/786317266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2894077640 con 0x7f2894075c60 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f2892ffd700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2894075c60 0x7f28940760e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f288800b4d0 tx=0x7f288800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.549+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2888011630 con 0x7f2894075c60 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2888011c80 con 0x7f2894075c60 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28941a4c40 con 0x7f2894075c60 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2888010da0 con 0x7f2894075c60 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28941a5180 con 0x7f2894075c60 2024-04-26T22:41:20.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f2888018020 con 0x7f2894075c60 2024-04-26T22:41:20.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2890ff9700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f28800393f0 0x7f288003b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:20.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f288804d140 con 0x7f2894075c60 2024-04-26T22:41:20.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2894104680 con 0x7f2894075c60 2024-04-26T22:41:20.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.550+0000 7f28937fe700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f28800393f0 0x7f288003b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:20.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.551+0000 7f28937fe700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f28800393f0 0x7f288003b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2884014a70 tx=0x7f2884021040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:20.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.556+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f288801f020 con 0x7f2894075c60 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: mon.a calling monitor election 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: mon.c calling monitor election 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T22:41:20.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: monmap epoch 2 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: last_changed 2024-04-26T22:41:15.214304+0000 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: min_mon_release 19 (squid) 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: election_strategy: 1 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: fsmap 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: mgrmap e13: x(active, since 50s) 2024-04-26T22:41:20.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: overall HEALTH_OK 2024-04-26T22:41:20.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:20.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:20.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:20.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:20.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:20.838 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.837+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f28941a5460 con 0x7f2894075c60 2024-04-26T22:41:20.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.838+0000 7f2890ff9700 1 -- 172.21.15.186:0/786317266 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f2888018310 con 0x7f2894075c60 2024-04-26T22:41:20.839 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:41:20.839 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","modified":"2024-04-26T22:41:15.214304Z","created":"2024-04-26T22:39:23.873268Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f28800393f0 msgr2=0x7f288003b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f28800393f0 0x7f288003b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2884014a70 tx=0x7f2884021040 comp rx=0 tx=0).stop 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2894075c60 msgr2=0x7f28940760e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2894075c60 0x7f28940760e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f288800b4d0 tx=0x7f288800b890 comp rx=0 tx=0).stop 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 shutdown_connections 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f28800393f0 0x7f288003b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f287c0035c0 0x7f2894075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 --2- 172.21.15.186:0/786317266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2894075c60 0x7f28940760e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 >> 172.21.15.186:0/786317266 conn(0x7f28940fea30 msgr2=0x7f28940ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:20.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 shutdown_connections 2024-04-26T22:41:20.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:20.840+0000 7f2899c57700 1 -- 172.21.15.186:0/786317266 wait complete. 2024-04-26T22:41:20.842 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-04-26T22:41:21.423 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:21 smithi186 ceph-mon[32774]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:21.423 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:21 smithi186 ceph-mon[32774]: Deploying daemon mon.b on smithi038 2024-04-26T22:41:21.423 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:21 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/786317266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:21.423 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:21 smithi001 ceph-mon[29435]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:21 smithi001 ceph-mon[29435]: Deploying daemon mon.b on smithi038 2024-04-26T22:41:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:21 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/786317266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:21.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:22.453 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T22:41:22.454 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T22:41:22.698 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:41:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:23 smithi186 ceph-mon[32774]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:23.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:23 smithi001 ceph-mon[29435]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:23.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:23 smithi038 podman[32412]: 2024-04-26 22:41:23.73952409 +0000 UTC m=+0.018706781 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 -- 172.21.15.186:0/1937782642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 msgr2=0x7f12b0103580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/1937782642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b0103580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009910 tx=0x7f12a4009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 -- 172.21.15.186:0/1937782642 shutdown_connections 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/1937782642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b0103580 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 -- 172.21.15.186:0/1937782642 >> 172.21.15.186:0/1937782642 conn(0x7f12b00fea30 msgr2=0x7f12b0100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 -- 172.21.15.186:0/1937782642 shutdown_connections 2024-04-26T22:41:24.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.096+0000 7f12b4e8b700 1 -- 172.21.15.186:0/1937782642 wait complete. 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12b4e8b700 1 Processor -- start 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12b4e8b700 1 -- start start 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12b4e8b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12b4e8b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12b4e8b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010d420 con 0x7f12b01031a0 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12b4e8b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010d2a0 con 0x7f12b019eca0 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12ae59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:24.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:55304/0 (socket says 172.21.15.186:55304) 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 -- 172.21.15.186:0/565036186 learned_addr learned my addr 172.21.15.186:0/565036186 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 -- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 msgr2=0x7f12b0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 -- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 msgr2=0x7f12b0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.097+0000 7f12add9b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f12ae59c700 1 -- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 msgr2=0x7f12b0198920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f12ae59c700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f12ae59c700 1 -- 172.21.15.186:0/565036186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b019f2e0 con 0x7f12b01031a0 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f12ae59c700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b019e760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f12a4014000 tx=0x7f12a4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f12a4005dc0 con 0x7f12b01031a0 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b0198e60 con 0x7f12b01031a0 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a400b750 con 0x7f12b01031a0 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f12a40039a0 con 0x7f12b01031a0 2024-04-26T22:41:24.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.098+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b01993f0 con 0x7f12b01031a0 2024-04-26T22:41:24.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.099+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b0104680 con 0x7f12b01031a0 2024-04-26T22:41:24.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.099+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f12a4029070 con 0x7f12b01031a0 2024-04-26T22:41:24.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.099+0000 7f129f7fe700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f12980393f0 0x7f129803b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:24.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.099+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f12a401e070 con 0x7f12b01031a0 2024-04-26T22:41:24.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.099+0000 7f12add9b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f12980393f0 0x7f129803b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:24.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.100+0000 7f12add9b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f12980393f0 0x7f129803b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009180 tx=0x7f12a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:24.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.105+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f12a4013070 con 0x7f12b01031a0 2024-04-26T22:41:24.125 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.124+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a402ae20 con 0x7f12b01031a0 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 podman[32412]: 2024-04-26 22:41:24.063219315 +0000 UTC m=+0.342402006 container init 714e571c3bf19cd348a06a6687b78c646b17659f9d457bb17b2f8cd65a3b660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 podman[32412]: 2024-04-26 22:41:24.066326473 +0000 UTC m=+0.345509165 container start 714e571c3bf19cd348a06a6687b78c646b17659f9d457bb17b2f8cd65a3b660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pidfile_write: ignore empty --pid-file 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: load: jerasure load: lrc 2024-04-26T22:41:24.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: RocksDB version: 7.9.2 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Git sha 0 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: DB SUMMARY 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: DB Session ID: 25JYVKIG0BP26GXCEIT4 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: CURRENT file: CURRENT 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.error_if_exists: 0 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.create_if_missing: 0 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.paranoid_checks: 1 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T22:41:24.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T22:41:24.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.env: 0x55897cd44cc0 2024-04-26T22:41:24.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T22:41:24.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.info_log: 0x55897f3f1d80 2024-04-26T22:41:24.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T22:41:24.355 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.statistics: (nil) 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.use_fsync: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_log_file_size: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.allow_fallocate: 1 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.use_direct_reads: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.db_log_dir: 2024-04-26T22:41:24.356 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.wal_dir: 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.write_buffer_manager: 0x55897f3c30e0 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T22:41:24.357 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.unordered_write: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.row_cache: None 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.wal_filter: None 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.two_write_queues: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T22:41:24.358 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.wal_compression: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.atomic_flush: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.log_readahead_size: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_background_jobs: 2 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_background_compactions: -1 2024-04-26T22:41:24.359 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_subcompactions: 1 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_open_files: -1 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_background_flushes: -1 2024-04-26T22:41:24.360 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Compression algorithms supported: 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kZSTD supported: 0 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kXpressCompression supported: 0 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kZlibCompression supported: 1 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kSnappyCompression supported: 1 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kLZ4Compression supported: 1 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: kBZip2Compression supported: 0 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.merge_operator: 2024-04-26T22:41:24.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_filter: None 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55897f3f1f40) 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_type: 0 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: data_block_index_type: 0 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_shortening: 1 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T22:41:24.362 INFO:journalctl@ceph.mon.b.smithi038.stdout: checksum: 4 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: no_block_cache: 0 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache: 0x55897f3f51f0 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_options: 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: capacity : 536870912 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: num_shard_bits : 4 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: strict_capacity_limit : 0 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_compressed: (nil) 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: persistent_cache: (nil) 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_size: 4096 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_size_deviation: 10 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_restart_interval: 16 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_block_restart_interval: 1 2024-04-26T22:41:24.363 INFO:journalctl@ceph.mon.b.smithi038.stdout: metadata_block_size: 4096 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: partition_filters: 0 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: use_delta_encoding: 1 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: filter_policy: bloomfilter 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: whole_key_filtering: 1 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: verify_compression: 0 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: format_version: 5 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: enable_index_compression: 1 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_align: 0 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: max_auto_readahead_size: 262144 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: prepopulate_block_cache: 0 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: initial_auto_readahead_size: 8192 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T22:41:24.364 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression: NoCompression 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.num_levels: 7 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T22:41:24.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T22:41:24.366 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T22:41:24.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T22:41:24.368 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.inplace_update_support: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.bloom_locality: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.max_successive_merges: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.ttl: 2592000 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T22:41:24.369 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.enable_blob_files: false 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.min_blob_size: 0 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0fb2588e-1db9-40e2-acec-d4f336e89efa 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714171284097430, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T22:41:24.370 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714171284097975, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714171284, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0fb2588e-1db9-40e2-acec-d4f336e89efa", "db_session_id": "25JYVKIG0BP26GXCEIT4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714171284098078, "job": 1, "event": "recovery_finished"} 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55897f512000 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: DB pointer 0x55897f4fe000 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** DB Stats ** 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T22:41:24.371 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-26T22:41:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Block cache BinnedLRUCache@0x55897f3f51f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T22:41:24.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: using public_addr v2:172.21.15.38:0/0 -> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: starting mon.b rank -1 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(???) e0 preinit fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).mds e1 new map 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout: e1 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout: legacy client fscid: -1 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout: No filesystems configured 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T22:41:24.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Added host smithi038 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Deploying cephadm binary to smithi186 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:41:24.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Added host smithi186 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.377 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2546668218' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2546668218' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T22:41:24.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Saving service mon spec with placement smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c;count:3 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Deploying daemon mon.c on smithi186 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/61768149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.a calling monitor election 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.c calling monitor election 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.380 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: monmap epoch 2 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: last_changed 2024-04-26T22:41:15.214304+0000 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: min_mon_release 19 (squid) 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: election_strategy: 1 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: fsmap 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mgrmap e13: x(active, since 50s) 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: overall HEALTH_OK 2024-04-26T22:41:24.381 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: Deploying daemon mon.b on smithi038 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/786317266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 ceph-mon[32433]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 bash[32412]: 714e571c3bf19cd348a06a6687b78c646b17659f9d457bb17b2f8cd65a3b660f 2024-04-26T22:41:24.382 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:24 smithi038 systemd[1]: Started Ceph mon.b for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:41:24.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:24.389+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f12b0199e50 con 0x7f12b01031a0 2024-04-26T22:41:29.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.139+0000 7f129f7fe700 1 -- 172.21.15.186:0/565036186 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1277 (secure 0 0 0) 0x7f12a4030090 con 0x7f12b01031a0 2024-04-26T22:41:29.142 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T22:41:29.143 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","modified":"2024-04-26T22:41:24.124750Z","created":"2024-04-26T22:39:23.873268Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-26T22:41:29.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f12980393f0 msgr2=0x7f129803b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:29.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f12980393f0 0x7f129803b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009180 tx=0x7f12a0008040 comp rx=0 tx=0).stop 2024-04-26T22:41:29.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 msgr2=0x7f12b019e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:29.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b019e760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f12a4014000 tx=0x7f12a4005030 comp rx=0 tx=0).stop 2024-04-26T22:41:29.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 shutdown_connections 2024-04-26T22:41:29.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f12980393f0 0x7f129803b8b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:29.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b01031a0 0x7f12b019e760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:29.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 --2- 172.21.15.186:0/565036186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12b019eca0 0x7f12b0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:29.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.142+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 >> 172.21.15.186:0/565036186 conn(0x7f12b00fea30 msgr2=0x7f12b0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:29.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.143+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 shutdown_connections 2024-04-26T22:41:29.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:29.143+0000 7f12b4e8b700 1 -- 172.21.15.186:0/565036186 wait complete. 2024-04-26T22:41:29.146 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: mon.a calling monitor election 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: mon.c calling monitor election 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/565036186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:29.452 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: mon.b calling monitor election 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: monmap epoch 3 2024-04-26T22:41:29.453 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: last_changed 2024-04-26T22:41:24.124750+0000 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: min_mon_release 19 (squid) 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: election_strategy: 1 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: fsmap 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: mgrmap e13: x(active, since 59s) 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T22:41:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:29.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: mon.a calling monitor election 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: mon.c calling monitor election 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/565036186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: mon.b calling monitor election 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: monmap epoch 3 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: last_changed 2024-04-26T22:41:24.124750+0000 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: min_mon_release 19 (squid) 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: election_strategy: 1 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: fsmap 2024-04-26T22:41:29.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:29.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: mgrmap e13: x(active, since 59s) 2024-04-26T22:41:29.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: overall HEALTH_OK 2024-04-26T22:41:29.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:29.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: mon.a calling monitor election 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: mon.c calling monitor election 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/565036186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: mon.b calling monitor election 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: monmap epoch 3 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: last_changed 2024-04-26T22:41:24.124750+0000 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: created 2024-04-26T22:39:23.873268+0000 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: min_mon_release 19 (squid) 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: election_strategy: 1 2024-04-26T22:41:29.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: fsmap 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: mgrmap e13: x(active, since 59s) 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: overall HEALTH_OK 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:29.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:29.699 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T22:41:29.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T22:41:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:41:30.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:30.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:30.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:30.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:30.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:41:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.666+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/2870392683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 msgr2=0x7f2f281023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.666+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/2870392683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f281023b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009910 tx=0x7f2f1c009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/2870392683 shutdown_connections 2024-04-26T22:41:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/2870392683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f281023b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/2870392683 >> 172.21.15.1:0/2870392683 conn(0x7f2f28077880 msgr2=0x7f2f28077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/2870392683 shutdown_connections 2024-04-26T22:41:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/2870392683 wait complete. 2024-04-26T22:41:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 Processor -- start 2024-04-26T22:41:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 -- start start 2024-04-26T22:41:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.667+0000 7f2f2f1d0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f2819e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2f1d0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2819ed40 0x7f2f281989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2f1d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f28198f00 0x7f2f281993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:31.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2f1d0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2810c200 con 0x7f2f28101fd0 2024-04-26T22:41:31.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2f1d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2810c080 con 0x7f2f28198f00 2024-04-26T22:41:31.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2f1d0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2810c380 con 0x7f2f2819ed40 2024-04-26T22:41:31.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f2819e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:31.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f2819e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42242/0 (socket says 172.21.15.1:42242) 2024-04-26T22:41:31.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 -- 172.21.15.1:0/3812306643 learned_addr learned my addr 172.21.15.1:0/3812306643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f27fff700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2819ed40 0x7f2f281989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2d76d700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f28198f00 0x7f2f281993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 -- 172.21.15.1:0/3812306643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2819ed40 msgr2=0x7f2f281989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2819ed40 0x7f2f281989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 -- 172.21.15.1:0/3812306643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f28198f00 msgr2=0x7f2f281993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f28198f00 0x7f2f281993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 -- 172.21.15.1:0/3812306643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f281a56d0 con 0x7f2f28101fd0 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2d76d700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f28198f00 0x7f2f281993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2cf6c700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f2819e800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c005b40 tx=0x7f2f1c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c026070 con 0x7f2f28101fd0 2024-04-26T22:41:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f1c00b6f0 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.668+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f281a5900 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.669+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c0038e0 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.669+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f281a5e90 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.670+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f2f1c029070 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.670+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f28064a80 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.670+0000 7f2f25ffb700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2f18039ae0 0x7f2f1803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.670+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2f1c04d500 con 0x7f2f28101fd0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.670+0000 7f2f27fff700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2f18039ae0 0x7f2f1803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.671+0000 7f2f27fff700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2f18039ae0 0x7f2f1803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009850 tx=0x7f2f14008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.675+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2f1c013070 con 0x7f2f28101fd0 2024-04-26T22:41:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:31.848 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:31.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:31.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:31.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.887+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f2f281a6120 con 0x7f2f28101fd0 2024-04-26T22:41:31.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.888+0000 7f2f25ffb700 1 -- 172.21.15.1:0/3812306643 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f2f1c030090 con 0x7f2f28101fd0 2024-04-26T22:41:31.889 INFO:teuthology.orchestra.run.smithi001.stdout:# minimal ceph.conf for bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:31.889 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-04-26T22:41:31.889 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:41:31.889 INFO:teuthology.orchestra.run.smithi001.stdout: mon_host = [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T22:41:31.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2f18039ae0 msgr2=0x7f2f1803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:31.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2f18039ae0 0x7f2f1803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009850 tx=0x7f2f14008040 comp rx=0 tx=0).stop 2024-04-26T22:41:31.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 msgr2=0x7f2f2819e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f2819e800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c005b40 tx=0x7f2f1c005030 comp rx=0 tx=0).stop 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 shutdown_connections 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2f18039ae0 0x7f2f1803bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f28198f00 0x7f2f281993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28101fd0 0x7f2f2819e800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.890+0000 7f2f2f1d0700 1 --2- 172.21.15.1:0/3812306643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2819ed40 0x7f2f281989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.891+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 >> 172.21.15.1:0/3812306643 conn(0x7f2f28077880 msgr2=0x7f2f2806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.891+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 shutdown_connections 2024-04-26T22:41:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:31.891+0000 7f2f2f1d0700 1 -- 172.21.15.1:0/3812306643 wait complete. 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:31.987 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:32.617 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T22:41:32.618 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:41:32.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T22:41:32.647 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:41:32.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:41:32.713 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:41:32.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T22:41:32.746 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:41:32.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:41:32.816 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:41:32.816 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Updating smithi001:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T22:41:32.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: Reconfiguring daemon mon.a on smithi001 2024-04-26T22:41:32.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:32 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3812306643' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:32.845 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:41:32.845 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T22:41:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T22:41:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T22:41:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T22:41:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Updating smithi001:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T22:41:32.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: Reconfiguring daemon mon.a on smithi001 2024-04-26T22:41:32.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:32 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3812306643' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Updating smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Updating smithi001:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Updating smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: Reconfiguring daemon mon.a on smithi001 2024-04-26T22:41:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:32 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3812306643' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:32.911 INFO:tasks.cephadm:Adding mgr.x on smithi001 2024-04-26T22:41:32.911 INFO:tasks.cephadm:Adding mgr.y on smithi038 2024-04-26T22:41:32.911 INFO:tasks.cephadm:Adding mgr.z on smithi186 2024-04-26T22:41:32.912 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi001=x;smithi038=y;smithi186=z' 2024-04-26T22:41:33.182 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:41:33.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:33 smithi001 ceph-mon[29435]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:33 smithi038 ceph-mon[32433]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:33 smithi186 ceph-mon[32774]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:34.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 -- 172.21.15.186:0/148337951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 msgr2=0x7fb7a81022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:34.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/148337951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81022a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009910 tx=0x7fb7a4009c20 comp rx=0 tx=0).stop 2024-04-26T22:41:34.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 -- 172.21.15.186:0/148337951 shutdown_connections 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/148337951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81022a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 -- 172.21.15.186:0/148337951 >> 172.21.15.186:0/148337951 conn(0x7fb7a80fd420 msgr2=0x7fb7a80ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 -- 172.21.15.186:0/148337951 shutdown_connections 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.591+0000 7fb7af2fd700 1 -- 172.21.15.186:0/148337951 wait complete. 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 Processor -- start 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 -- start start 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7a8198c10 0x7fb7a819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 0x7fb7a81a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8107d40 con 0x7fb7a8101e80 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8107bc0 con 0x7fb7a8198c10 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7af2fd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8107ec0 con 0x7fb7a8199090 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7ad89a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 0x7fb7a81a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:34.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7ad89a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 0x7fb7a81a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34042/0 (socket says 172.21.15.186:34042) 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7ad89a700 1 -- 172.21.15.186:0/4182111679 learned_addr learned my addr 172.21.15.186:0/4182111679 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7ad099700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.592+0000 7fb7ac898700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7a8198c10 0x7fb7a819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad89a700 1 -- 172.21.15.186:0/4182111679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7a8198c10 msgr2=0x7fb7a819dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad89a700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7a8198c10 0x7fb7a819dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad89a700 1 -- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 msgr2=0x7fb7a81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad89a700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad89a700 1 -- 172.21.15.186:0/4182111679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a81a0ae0 con 0x7fb7a8199090 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad099700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7ad89a700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 0x7fb7a81a0210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000ba50 tx=0x7fb7a000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0019070 con 0x7fb7a8199090 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a0014440 con 0x7fb7a8199090 2024-04-26T22:41:34.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a000ce60 con 0x7fb7a8199090 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a81a0dd0 con 0x7fb7a8199090 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.593+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a81a12b0 con 0x7fb7a8199090 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.594+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fb7a001c070 con 0x7fb7a8199090 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.594+0000 7fb79e7fc700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb794039ae0 0x7fb79403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.594+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a8064a80 con 0x7fb7a8199090 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.594+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb7a004be40 con 0x7fb7a8199090 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.594+0000 7fb7ad099700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb794039ae0 0x7fb79403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:34.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.595+0000 7fb7ad099700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb794039ae0 0x7fb79403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4005850 tx=0x7fb7a401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:34.600 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.599+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb7a001b2c0 con 0x7fb7a8199090 2024-04-26T22:41:34.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.806+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}) -- 0x7fb7a80638f0 con 0x7fb794039ae0 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: Reconfiguring mon.b (monmap changed)... 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:34.807 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:34.808 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:34 smithi186 ceph-mon[32774]: Reconfiguring daemon mon.b on smithi038 2024-04-26T22:41:34.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.812+0000 7fb79e7fc700 1 -- 172.21.15.186:0/4182111679 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb7a80638f0 con 0x7fb794039ae0 2024-04-26T22:41:34.813 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-04-26T22:41:34.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb794039ae0 msgr2=0x7fb79403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:34.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb794039ae0 0x7fb79403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4005850 tx=0x7fb7a401a040 comp rx=0 tx=0).stop 2024-04-26T22:41:34.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 msgr2=0x7fb7a81a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:34.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 0x7fb7a81a0210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000ba50 tx=0x7fb7a000bd60 comp rx=0 tx=0).stop 2024-04-26T22:41:34.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 shutdown_connections 2024-04-26T22:41:34.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb794039ae0 0x7fb79403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7a8199090 0x7fb7a81a0210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7a8101e80 0x7fb7a81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 --2- 172.21.15.186:0/4182111679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7a8198c10 0x7fb7a819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 >> 172.21.15.186:0/4182111679 conn(0x7fb7a80fd420 msgr2=0x7fb7a806b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 shutdown_connections 2024-04-26T22:41:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:41:34.815+0000 7fb7af2fd700 1 -- 172.21.15.186:0/4182111679 wait complete. 2024-04-26T22:41:35.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:35.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: Reconfiguring mon.b (monmap changed)... 2024-04-26T22:41:35.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:35.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:35.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:35.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:34 smithi001 ceph-mon[29435]: Reconfiguring daemon mon.b on smithi038 2024-04-26T22:41:35.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:35.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: Reconfiguring mon.b (monmap changed)... 2024-04-26T22:41:35.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:35.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:35.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:35.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:34 smithi038 ceph-mon[32433]: Reconfiguring daemon mon.b on smithi038 2024-04-26T22:41:35.433 DEBUG:teuthology.orchestra.run.smithi038:mgr.y> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T22:41:35.435 DEBUG:teuthology.orchestra.run.smithi186:mgr.z> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T22:41:35.437 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T22:41:35.438 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:41:35.438 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T22:41:35.457 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T22:41:35.458 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 22:40:39.699805315 +0000 2024-04-26T22:41:35.516 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 22:36:09.222710479 +0000 2024-04-26T22:41:35.517 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 22:36:09.222710479 +0000 2024-04-26T22:41:35.517 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 22:36:09.222710479 +0000 2024-04-26T22:41:35.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T22:41:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T22:41:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T22:41:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000114646 s, 4.5 MB/s 2024-04-26T22:41:35.586 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T22:41:35.645 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-04-26T22:41:35.702 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 22:40:39.737804866 +0000 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 22:36:09.561706338 +0000 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 22:36:09.561706338 +0000 2024-04-26T22:41:35.703 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 22:36:09.561706338 +0000 2024-04-26T22:41:35.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T22:41:35.768 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T22:41:35.768 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T22:41:35.768 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000107138 s, 4.8 MB/s 2024-04-26T22:41:35.769 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T22:41:35.827 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-04-26T22:41:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: Saving service mgr spec with placement smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: Reconfiguring mon.c (monmap changed)... 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:35 smithi001 ceph-mon[29435]: Reconfiguring daemon mon.c on smithi186 2024-04-26T22:41:35.862 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 22:40:39.775804417 +0000 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 22:36:09.893702283 +0000 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 22:36:09.893702283 +0000 2024-04-26T22:41:35.863 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 22:36:09.893702283 +0000 2024-04-26T22:41:35.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T22:41:35.927 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T22:41:35.927 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T22:41:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000120314 s, 4.3 MB/s 2024-04-26T22:41:35.929 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T22:41:35.986 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-04-26T22:41:36.042 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T22:41:36.042 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:36.042 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-26T22:41:36.043 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:36.043 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:36.043 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 22:40:39.812803980 +0000 2024-04-26T22:41:36.043 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 22:36:10.242698020 +0000 2024-04-26T22:41:36.043 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 22:36:10.242698020 +0000 2024-04-26T22:41:36.043 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 22:36:10.242698020 +0000 2024-04-26T22:41:36.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: Saving service mgr spec with placement smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: Reconfiguring mon.c (monmap changed)... 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:36.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:35 smithi186 ceph-mon[32774]: Reconfiguring daemon mon.c on smithi186 2024-04-26T22:41:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: Saving service mgr spec with placement smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T22:41:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:36.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: Reconfiguring mon.c (monmap changed)... 2024-04-26T22:41:36.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T22:41:36.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T22:41:36.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:36.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:35 smithi038 ceph-mon[32433]: Reconfiguring daemon mon.c on smithi186 2024-04-26T22:41:36.106 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T22:41:36.106 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T22:41:36.106 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000119258 s, 4.3 MB/s 2024-04-26T22:41:36.107 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T22:41:36.164 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T22:41:36.164 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T22:41:36.182 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T22:41:36.183 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-26T22:41:36.240 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T22:41:36.240 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:36.240 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T22:41:36.240 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:36.241 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:36.241 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 22:41:23.520441184 +0000 2024-04-26T22:41:36.241 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 22:36:17.253096692 +0000 2024-04-26T22:41:36.241 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 22:36:17.253096692 +0000 2024-04-26T22:41:36.241 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 22:36:17.253096692 +0000 2024-04-26T22:41:36.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T22:41:36.307 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T22:41:36.307 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T22:41:36.307 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00032268 s, 1.6 MB/s 2024-04-26T22:41:36.308 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T22:41:36.367 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-26T22:41:36.423 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 22:41:23.521441166 +0000 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 22:36:17.620089554 +0000 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 22:36:17.620089554 +0000 2024-04-26T22:41:36.424 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 22:36:17.620089554 +0000 2024-04-26T22:41:36.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T22:41:36.489 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T22:41:36.489 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T22:41:36.489 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000282173 s, 1.8 MB/s 2024-04-26T22:41:36.490 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T22:41:36.548 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-26T22:41:36.605 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T22:41:36.605 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 22:41:23.521441166 +0000 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 22:36:17.978082591 +0000 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 22:36:17.978082591 +0000 2024-04-26T22:41:36.606 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 22:36:17.978082591 +0000 2024-04-26T22:41:36.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T22:41:36.670 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T22:41:36.671 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T22:41:36.671 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000271566 s, 1.9 MB/s 2024-04-26T22:41:36.672 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T22:41:36.729 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-26T22:41:36.785 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T22:41:36.785 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:36.785 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-26T22:41:36.785 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:36.785 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:36.785 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 22:41:23.522441148 +0000 2024-04-26T22:41:36.786 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 22:36:18.337075609 +0000 2024-04-26T22:41:36.786 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 22:36:18.337075609 +0000 2024-04-26T22:41:36.786 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 22:36:18.337075609 +0000 2024-04-26T22:41:36.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T22:41:36.850 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T22:41:36.850 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T22:41:36.850 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000363903 s, 1.4 MB/s 2024-04-26T22:41:36.852 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T22:41:36.909 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T22:41:36.909 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T22:41:36.926 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T22:41:36.926 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-26T22:41:36.983 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T22:41:36.983 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:36.983 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T22:41:36.984 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:36.984 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:36.984 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 22:41:14.678103570 +0000 2024-04-26T22:41:36.984 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 22:36:20.531403710 +0000 2024-04-26T22:41:36.984 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 22:36:20.531403710 +0000 2024-04-26T22:41:36.984 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 22:36:20.531403710 +0000 2024-04-26T22:41:36.984 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T22:41:37.048 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T22:41:37.048 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T22:41:37.048 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000273057 s, 1.9 MB/s 2024-04-26T22:41:37.049 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T22:41:37.106 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-26T22:41:37.161 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T22:41:37.161 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 22:41:14.678103570 +0000 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 22:36:20.874397189 +0000 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 22:36:20.874397189 +0000 2024-04-26T22:41:37.162 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 22:36:20.874397189 +0000 2024-04-26T22:41:37.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T22:41:37.224 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T22:41:37.224 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T22:41:37.224 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000196072 s, 2.6 MB/s 2024-04-26T22:41:37.225 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T22:41:37.281 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-26T22:41:37.336 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T22:41:37.336 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:37.336 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-26T22:41:37.336 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:37.336 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:37.336 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 22:41:14.679103552 +0000 2024-04-26T22:41:37.337 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 22:36:21.227390478 +0000 2024-04-26T22:41:37.337 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 22:36:21.227390478 +0000 2024-04-26T22:41:37.337 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 22:36:21.227390478 +0000 2024-04-26T22:41:37.337 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T22:41:37.398 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T22:41:37.399 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T22:41:37.399 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000201004 s, 2.5 MB/s 2024-04-26T22:41:37.400 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T22:41:37.456 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-26T22:41:37.511 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T22:41:37.511 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:41:37.511 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-26T22:41:37.511 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:41:37.512 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:41:37.512 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 22:41:14.679103552 +0000 2024-04-26T22:41:37.512 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 22:36:21.568383995 +0000 2024-04-26T22:41:37.512 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 22:36:21.568383995 +0000 2024-04-26T22:41:37.512 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 22:36:21.568383995 +0000 2024-04-26T22:41:37.512 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T22:41:37.576 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T22:41:37.576 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T22:41:37.576 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000193165 s, 2.7 MB/s 2024-04-26T22:41:37.577 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T22:41:37.633 INFO:tasks.cephadm:Deploying osd.0 on smithi001 with /dev/vg_nvme/lv_4... 2024-04-26T22:41:37.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T22:41:37.655 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:37 smithi001 ceph-mon[29435]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:37.655 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:37.655 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:37.655 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:37 smithi186 ceph-mon[32774]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:37 smithi038 ceph-mon[32433]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:37.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:37.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:41:38.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:38.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:38.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:38.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:38.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T22:41:38.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:38.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:38.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:38 smithi001 ceph-mon[29435]: Deploying daemon mgr.z on smithi186 2024-04-26T22:41:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:38.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T22:41:38.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:38.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:38.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:38 smithi186 ceph-mon[32774]: Deploying daemon mgr.z on smithi186 2024-04-26T22:41:38.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:38.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:38.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:38.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:38.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T22:41:38.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:38.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:38.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:38 smithi038 ceph-mon[32433]: Deploying daemon mgr.z on smithi186 2024-04-26T22:41:39.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:39 smithi001 ceph-mon[29435]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:39.664 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:39 smithi186 ceph-mon[32774]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T22:41:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T22:41:39.842 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T22:41:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T22:41:39.842 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00840596 s, 1.2 GB/s 2024-04-26T22:41:39.842 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T22:41:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:39 smithi038 ceph-mon[32433]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:39.931 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:39 smithi186 systemd[1]: Started Ceph mgr.z for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:41:40.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:39.930+0000 7f6ef9149200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T22:41:40.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:40.115+0000 7f6ef9149200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:41:40.219 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:40.218+0000 7f6ef9149200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T22:41:40.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_4 2024-04-26T22:41:40.872 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:41:40.914 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:40.526+0000 7f6ef9149200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T22:41:40.914 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:40.609+0000 7f6ef9149200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T22:41:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:40 smithi038 ceph-mon[32433]: Deploying daemon mgr.y on smithi038 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:41.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:40 smithi001 ceph-mon[29435]: Deploying daemon mgr.y on smithi038 2024-04-26T22:41:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:41.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T22:41:41.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:41.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:41.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:40 smithi186 ceph-mon[32774]: Deploying daemon mgr.y on smithi038 2024-04-26T22:41:41.985 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:41.831+0000 7f6ef9149200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T22:41:42.066 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:41 smithi038 systemd[1]: Starting Ceph mgr.y for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:41:42.352 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 podman[33798]: 2024-04-26 22:41:42.066290503 +0000 UTC m=+0.109473603 container create 5c25694dee7b809ac6f1d30ae7c4ae49650ad1ac389715b39e8740e09f1f8419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T22:41:42.352 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 podman[33798]: 2024-04-26 22:41:41.975435854 +0000 UTC m=+0.018618953 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:41:42.352 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 podman[33798]: 2024-04-26 22:41:42.277974664 +0000 UTC m=+0.321157763 container init 5c25694dee7b809ac6f1d30ae7c4ae49650ad1ac389715b39e8740e09f1f8419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T22:41:42.352 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 podman[33798]: 2024-04-26 22:41:42.280720427 +0000 UTC m=+0.323903530 container start 5c25694dee7b809ac6f1d30ae7c4ae49650ad1ac389715b39e8740e09f1f8419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T22:41:42.352 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 ceph-mgr[33817]: -- 172.21.15.38:0/3980682340 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8eec525a0 con 0x55a8eec26c00 2024-04-26T22:41:42.485 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:42.008+0000 7f6ef9149200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T22:41:42.485 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:42.185+0000 7f6ef9149200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T22:41:42.485 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:42.484+0000 7f6ef9149200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T22:41:42.637 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 bash[33798]: 5c25694dee7b809ac6f1d30ae7c4ae49650ad1ac389715b39e8740e09f1f8419 2024-04-26T22:41:42.637 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 systemd[1]: Started Ceph mgr.y for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:41:42.637 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:42.445+0000 7ff6db29d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T22:41:42.637 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:42.637+0000 7ff6db29d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:41:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 -- 172.21.15.1:0/3021647317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe790105040 msgr2=0x7fe79010b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 --2- 172.21.15.1:0/3021647317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe790105040 0x7fe79010b8f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe78800b0d0 tx=0x7fe78800b3e0 comp rx=0 tx=0).stop 2024-04-26T22:41:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 -- 172.21.15.1:0/3021647317 shutdown_connections 2024-04-26T22:41:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 --2- 172.21.15.1:0/3021647317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe790105040 0x7fe79010b8f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 --2- 172.21.15.1:0/3021647317 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901034a0 0x7fe7901038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 --2- 172.21.15.1:0/3021647317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7901046a0 0x7fe790104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.645+0000 7fe797987700 1 -- 172.21.15.1:0/3021647317 >> 172.21.15.1:0/3021647317 conn(0x7fe7900fea40 msgr2=0x7fe790100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 -- 172.21.15.1:0/3021647317 shutdown_connections 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 -- 172.21.15.1:0/3021647317 wait complete. 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 Processor -- start 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 -- start start 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 0x7fe790199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901046a0 0x7fe790199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.646+0000 7fe797987700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe790105040 0x7fe7901a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe797987700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe79010ef70 con 0x7fe7901034a0 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe797987700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe79010edf0 con 0x7fe7901046a0 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe797987700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe79010f0f0 con 0x7fe790105040 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 0x7fe790199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 0x7fe790199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44938/0 (socket says 172.21.15.1:44938) 2024-04-26T22:41:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 -- 172.21.15.1:0/2550057103 learned_addr learned my addr 172.21.15.1:0/2550057103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe794f22700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901046a0 0x7fe790199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795f24700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe790105040 0x7fe7901a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 -- 172.21.15.1:0/2550057103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe790105040 msgr2=0x7fe7901a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe790105040 0x7fe7901a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 -- 172.21.15.1:0/2550057103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901046a0 msgr2=0x7fe790199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901046a0 0x7fe790199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 -- 172.21.15.1:0/2550057103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7901a11c0 con 0x7fe7901034a0 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe794f22700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901046a0 0x7fe790199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe795723700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 0x7fe790199560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00e9a0 tx=0x7fe78c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c004220 con 0x7fe7901034a0 2024-04-26T22:41:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe78c0043c0 con 0x7fe7901034a0 2024-04-26T22:41:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7901a14b0 con 0x7fe7901034a0 2024-04-26T22:41:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c003cc0 con 0x7fe7901034a0 2024-04-26T22:41:42.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.647+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7901a19f0 con 0x7fe7901034a0 2024-04-26T22:41:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.648+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fe78c018430 con 0x7fe7901034a0 2024-04-26T22:41:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.648+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79006a110 con 0x7fe7901034a0 2024-04-26T22:41:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.648+0000 7fe7867fc700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe77c039890 0x7fe77c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.648+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe78c04cc30 con 0x7fe7901034a0 2024-04-26T22:41:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.649+0000 7fe794f22700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe77c039890 0x7fe77c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:42.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.649+0000 7fe794f22700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe77c039890 0x7fe77c03bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe780007fd0 tx=0x7fe7800073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:42.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.654+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe78c017db0 con 0x7fe7901034a0 2024-04-26T22:41:42.735 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:42.578+0000 7f6ef9149200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T22:41:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:42.874+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe790063920 con 0x7fe77c039890 2024-04-26T22:41:43.075 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:42.745+0000 7ff6db29d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T22:41:43.075 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:43.075+0000 7ff6db29d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T22:41:43.235 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:42.793+0000 7f6ef9149200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T22:41:43.235 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:42.994+0000 7f6ef9149200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:41:43.351 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:43.173+0000 7ff6db29d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:41:43.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:43.735 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:43.639+0000 7f6ef9149200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T22:41:43.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:41:43.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:41:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.376 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:44.076+0000 7f6ef9149200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T22:41:44.377 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:44.176+0000 7f6ef9149200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T22:41:44.377 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:44.376+0000 7f6ef9149200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T22:41:44.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:44.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:44.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.738 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:44.738 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.738 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.738 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:44.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T22:41:44.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T22:41:44.809 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:44.809 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:44.457+0000 7ff6db29d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T22:41:44.809 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:44.632+0000 7ff6db29d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T22:41:44.809 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:44.807+0000 7ff6db29d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T22:41:45.235 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:44.914+0000 7f6ef9149200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T22:41:45.235 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:45.005+0000 7f6ef9149200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T22:41:45.427 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:45.104+0000 7ff6db29d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T22:41:45.427 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:45.200+0000 7ff6db29d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T22:41:45.427 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:45.427+0000 7ff6db29d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T22:41:45.596 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-mon[32774]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:45.596 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-mon[32774]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T22:41:45.596 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-mon[32774]: Reconfiguring daemon mgr.x on smithi001 2024-04-26T22:41:45.596 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:45.597 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:45.249+0000 7f6ef9149200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T22:41:45.597 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:45.382+0000 7f6ef9149200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T22:41:45.597 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:45.478+0000 7f6ef9149200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T22:41:45.597 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:45.595+0000 7f6ef9149200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T22:41:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:45 smithi001 ceph-mon[29435]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:45 smithi001 ceph-mon[29435]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T22:41:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:45 smithi001 ceph-mon[29435]: Reconfiguring daemon mgr.x on smithi001 2024-04-26T22:41:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:45.851 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:45.625+0000 7ff6db29d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T22:41:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-mon[32433]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-mon[32433]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T22:41:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-mon[32433]: Reconfiguring daemon mgr.x on smithi001 2024-04-26T22:41:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:46.235 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:45.903+0000 7f6ef9149200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T22:41:46.601 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:46.272+0000 7ff6db29d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T22:41:46.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/382650459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5213229d-410b-4bd5-9fb0-fb58a13e3322"}]: dispatch 2024-04-26T22:41:46.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/382650459' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5213229d-410b-4bd5-9fb0-fb58a13e3322"}]': finished 2024-04-26T22:41:46.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-mon[32433]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T22:41:46.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:41:46.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1916121484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:41:46.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/382650459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5213229d-410b-4bd5-9fb0-fb58a13e3322"}]: dispatch 2024-04-26T22:41:46.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/382650459' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5213229d-410b-4bd5-9fb0-fb58a13e3322"}]': finished 2024-04-26T22:41:46.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-mon[32774]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T22:41:46.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:41:46.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1916121484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:41:46.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:46.508+0000 7f6ef9149200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T22:41:46.726 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 22:41:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z[34630]: 2024-04-26T22:41:46.724+0000 7f6ef9149200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T22:41:46.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/382650459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5213229d-410b-4bd5-9fb0-fb58a13e3322"}]: dispatch 2024-04-26T22:41:46.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/382650459' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5213229d-410b-4bd5-9fb0-fb58a13e3322"}]': finished 2024-04-26T22:41:46.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:46 smithi001 ceph-mon[29435]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T22:41:46.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:46 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:41:46.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1916121484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:41:47.021 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:46.712+0000 7ff6db29d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T22:41:47.021 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:46.812+0000 7ff6db29d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T22:41:47.021 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:47.021+0000 7ff6db29d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T22:41:47.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:47.580+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 14) ==== 92601+0+0 (secure 0 0 0) 0x7fe78c021520 con 0x7fe7901034a0 2024-04-26T22:41:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:47 smithi001 ceph-mon[29435]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:47.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:47 smithi001 ceph-mon[29435]: Standby manager daemon z started 2024-04-26T22:41:47.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:47 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T22:41:47.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:47 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T22:41:47.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:47 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T22:41:47.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:47 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T22:41:47.851 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:47.570+0000 7ff6db29d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T22:41:47.851 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:47.665+0000 7ff6db29d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T22:41:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-mon[32433]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-mon[32433]: Standby manager daemon z started 2024-04-26T22:41:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T22:41:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T22:41:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T22:41:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T22:41:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:47 smithi186 ceph-mon[32774]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:47 smithi186 ceph-mon[32774]: Standby manager daemon z started 2024-04-26T22:41:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:47 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T22:41:47.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:47 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T22:41:47.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:47 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T22:41:47.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:47 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.186:0/2921970014' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T22:41:48.270 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:47.916+0000 7ff6db29d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T22:41:48.271 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:48.053+0000 7ff6db29d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T22:41:48.271 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:48.150+0000 7ff6db29d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T22:41:48.271 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:48.269+0000 7ff6db29d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T22:41:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:48 smithi001 ceph-mon[29435]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-26T22:41:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T22:41:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:48 smithi001 ceph-mon[29435]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:48.851 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:48.581+0000 7ff6db29d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T22:41:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-mon[32433]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-26T22:41:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T22:41:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:48 smithi038 ceph-mon[32433]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:48.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:48 smithi186 ceph-mon[32774]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-26T22:41:48.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T22:41:48.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:48 smithi186 ceph-mon[32774]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:49.585 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:49.204+0000 7ff6db29d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T22:41:49.585 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y[33813]: 2024-04-26T22:41:49.426+0000 7ff6db29d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T22:41:49.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:41:49.592+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7fe78c020cc0 con 0x7fe7901034a0 2024-04-26T22:41:49.824 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:49 smithi001 ceph-mon[29435]: Standby manager daemon y started 2024-04-26T22:41:49.824 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:49 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T22:41:49.825 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:49 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T22:41:49.825 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:49 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T22:41:49.825 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:49 smithi001 ceph-mon[29435]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T22:41:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-mon[32433]: Standby manager daemon y started 2024-04-26T22:41:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T22:41:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T22:41:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T22:41:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:49 smithi038 ceph-mon[32433]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T22:41:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:49 smithi186 ceph-mon[32774]: Standby manager daemon y started 2024-04-26T22:41:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:49 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T22:41:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:49 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T22:41:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:49 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T22:41:49.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:49 smithi186 ceph-mon[32774]: from='mgr.? 172.21.15.38:0/1941330532' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T22:41:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:50 smithi038 ceph-mon[32433]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-26T22:41:50.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T22:41:50.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:50 smithi038 ceph-mon[32433]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:50.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:50 smithi001 ceph-mon[29435]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-26T22:41:50.931 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T22:41:50.931 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:50 smithi001 ceph-mon[29435]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:50.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:50 smithi186 ceph-mon[32774]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-26T22:41:50.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T22:41:50.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:50 smithi186 ceph-mon[32774]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:52.145 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.145 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.145 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:52.145 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:52.145 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:52.145 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:52.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:52.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:52.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:52.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:52.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:52.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:52.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:53.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:53 smithi001 ceph-mon[29435]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:53 smithi038 ceph-mon[32433]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:53.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:53 smithi186 ceph-mon[32774]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:54.338 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T22:41:54.338 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:54.338 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:54 smithi001 ceph-mon[29435]: Deploying daemon osd.0 on smithi001 2024-04-26T22:41:54.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T22:41:54.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:54.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:54 smithi038 ceph-mon[32433]: Deploying daemon osd.0 on smithi001 2024-04-26T22:41:54.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T22:41:54.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:54.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:54 smithi186 ceph-mon[32774]: Deploying daemon osd.0 on smithi001 2024-04-26T22:41:55.253 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:55 smithi001 ceph-mon[29435]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:55 smithi038 ceph-mon[32433]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:55 smithi186 ceph-mon[32774]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:57.236 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:57 smithi001 ceph-mon[29435]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:57.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:57 smithi038 ceph-mon[32433]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:57.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:57 smithi186 ceph-mon[32774]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:59 smithi001 ceph-mon[29435]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:59.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:59.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:41:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:59 smithi038 ceph-mon[32433]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:41:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:59 smithi186 ceph-mon[32774]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:41:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:41:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:41:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:01.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:01 smithi001 ceph-mon[29435]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:01 smithi038 ceph-mon[32433]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:01 smithi186 ceph-mon[32774]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:02.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.430+0000 7fe7867fc700 1 -- 172.21.15.1:0/2550057103 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe790063920 con 0x7fe77c039890 2024-04-26T22:42:02.431 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 0 on host 'smithi001' 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe77c039890 msgr2=0x7fe77c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe77c039890 0x7fe77c03bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe780007fd0 tx=0x7fe7800073d0 comp rx=0 tx=0).stop 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 msgr2=0x7fe790199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 0x7fe790199560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00e9a0 tx=0x7fe78c00ecb0 comp rx=0 tx=0).stop 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 shutdown_connections 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe77c039890 0x7fe77c03bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe790105040 0x7fe7901a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7901034a0 0x7fe790199560 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 --2- 172.21.15.1:0/2550057103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7901046a0 0x7fe790199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 >> 172.21.15.1:0/2550057103 conn(0x7fe7900fea40 msgr2=0x7fe7901079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 shutdown_connections 2024-04-26T22:42:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:02.432+0000 7fe797987700 1 -- 172.21.15.1:0/2550057103 wait complete. 2024-04-26T22:42:03.084 DEBUG:teuthology.orchestra.run.smithi001:osd.0> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T22:42:03.086 INFO:tasks.cephadm:Deploying osd.1 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-26T22:42:03.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T22:42:03.343 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:42:03.685 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:03 smithi038 ceph-mon[32433]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:03.685 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.686 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.686 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:03 smithi038 ceph-mon[32433]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T22:42:03.686 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.686 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:03 smithi186 ceph-mon[32774]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:03.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:03 smithi186 ceph-mon[32774]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T22:42:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:03 smithi001 ceph-mon[29435]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:03.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:03 smithi001 ceph-mon[29435]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T22:42:03.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:03.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:04.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:04 smithi038 ceph-mon[32433]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T22:42:04.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:04 smithi038 ceph-mon[32433]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T22:42:04.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:04 smithi038 ceph-mon[32433]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:42:04.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:04 smithi186 ceph-mon[32774]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T22:42:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:04 smithi186 ceph-mon[32774]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T22:42:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:04 smithi186 ceph-mon[32774]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:42:04.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:04.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:04 smithi001 ceph-mon[29435]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T22:42:04.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:04 smithi001 ceph-mon[29435]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T22:42:04.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:04 smithi001 ceph-mon[29435]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:42:04.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:04.819 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0[42090]: 2024-04-26T22:42:04.409+0000 7fee0ebde700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T22:42:04.819 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0[42090]: 2024-04-26T22:42:04.412+0000 7fee0a9d1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:42:05.143 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T22:42:05.143 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T22:42:05.144 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T22:42:05.144 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T22:42:05.144 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0159763 s, 656 MB/s 2024-04-26T22:42:05.144 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T22:42:05.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:05 smithi186 ceph-mon[32774]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:05.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:05 smithi186 ceph-mon[32774]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:42:05.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:05 smithi186 ceph-mon[32774]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T22:42:05.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:05.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:05.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:05 smithi038 ceph-mon[32433]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:05.837 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:05 smithi038 ceph-mon[32433]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:42:05.837 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:05 smithi038 ceph-mon[32433]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T22:42:05.837 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:05.837 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:05 smithi001 ceph-mon[29435]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T22:42:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:05 smithi001 ceph-mon[29435]: from='osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:42:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:05 smithi001 ceph-mon[29435]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T22:42:05.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:05.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:05.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-26T22:42:06.118 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:42:06.563 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:06 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:42:06.563 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:06 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:42:06.564 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:06 smithi038 ceph-mon[32433]: osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] boot 2024-04-26T22:42:06.564 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:06 smithi038 ceph-mon[32433]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T22:42:06.564 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:06.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:06 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:42:06.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:06 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:42:06.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:06 smithi001 ceph-mon[29435]: osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] boot 2024-04-26T22:42:06.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:06 smithi001 ceph-mon[29435]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T22:42:06.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:06.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:06 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:42:06.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:06 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:42:06.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:06 smithi186 ceph-mon[32774]: osd.0 [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] boot 2024-04-26T22:42:06.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:06 smithi186 ceph-mon[32774]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T22:42:06.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:07.448 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:07.449 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.449 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:07.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 -- 172.21.15.38:0/382197068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 msgr2=0x7fc5981027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:07.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/382197068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc5981027e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc5880099c0 tx=0x7fc588009cd0 comp rx=0 tx=0).stop 2024-04-26T22:42:07.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 -- 172.21.15.38:0/382197068 shutdown_connections 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/382197068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc598103f60 0x7fc59810a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/382197068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc5981027e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/382197068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5981035c0 0x7fc598103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 -- 172.21.15.38:0/382197068 >> 172.21.15.38:0/382197068 conn(0x7fc5980fd980 msgr2=0x7fc5980ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 -- 172.21.15.38:0/382197068 shutdown_connections 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.638+0000 7fc59e7f6700 1 -- 172.21.15.38:0/382197068 wait complete. 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 Processor -- start 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 -- start start 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc598199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5981035c0 0x7fc598199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:07.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc598103f60 0x7fc5981a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810de60 con 0x7fc5981035c0 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810dce0 con 0x7fc5981023c0 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc59e7f6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810dfe0 con 0x7fc598103f60 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.639+0000 7fc597fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc598199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc598199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54310/0 (socket says 172.21.15.38:54310) 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 -- 172.21.15.38:0/3095213406 learned_addr learned my addr 172.21.15.38:0/3095213406 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc5977fe700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5981035c0 0x7fc598199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc59cd93700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc598103f60 0x7fc5981a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 -- 172.21.15.38:0/3095213406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc598103f60 msgr2=0x7fc5981a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc598103f60 0x7fc5981a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 -- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5981035c0 msgr2=0x7fc598199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:07.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5981035c0 0x7fc598199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 -- 172.21.15.38:0/3095213406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5981a11d0 con 0x7fc5981023c0 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc597fff700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc598199570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc58800fa00 tx=0x7fc58800fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588027070 con 0x7fc5981023c0 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc588005940 con 0x7fc5981023c0 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5981a1460 con 0x7fc5981023c0 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5880173f0 con 0x7fc5981023c0 2024-04-26T22:42:07.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.640+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5981a1940 con 0x7fc5981023c0 2024-04-26T22:42:07.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.641+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59806a110 con 0x7fc5981023c0 2024-04-26T22:42:07.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.643+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7fc58802a070 con 0x7fc5981023c0 2024-04-26T22:42:07.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.643+0000 7fc5957fa700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5800a3bb0 0x7fc5800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:07.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.643+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fc5880cf600 con 0x7fc5981023c0 2024-04-26T22:42:07.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.643+0000 7fc5977fe700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5800a3bb0 0x7fc5800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:07.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.644+0000 7fc5977fe700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5800a3bb0 0x7fc5800a6070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00ac50 tx=0x7fc58c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:42:07.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.648+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc58805f060 con 0x7fc5981023c0 2024-04-26T22:42:07.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:07.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:07.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:07.870+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc5980ffb00 con 0x7fc5800a3bb0 2024-04-26T22:42:08.722 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi001 2024-04-26T22:42:08.722 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi001 to 17093M 2024-04-26T22:42:08.722 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:08.722 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:08.723 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:08.723 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:08.723 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:08.723 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi001 2024-04-26T22:42:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi001 to 17093M 2024-04-26T22:42:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:08.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:08.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi001 2024-04-26T22:42:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi001 to 17093M 2024-04-26T22:42:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:09.789 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:09 smithi038 ceph-mon[32433]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:09.789 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:09 smithi038 ceph-mon[32433]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:09 smithi001 ceph-mon[29435]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:09 smithi001 ceph-mon[29435]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:09 smithi186 ceph-mon[32774]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:09 smithi186 ceph-mon[32774]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/4240156806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]: dispatch 2024-04-26T22:42:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]: dispatch 2024-04-26T22:42:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]': finished 2024-04-26T22:42:11.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T22:42:11.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:11.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:11 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/3134809103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:42:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/4240156806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]: dispatch 2024-04-26T22:42:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]: dispatch 2024-04-26T22:42:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]': finished 2024-04-26T22:42:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T22:42:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:11.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:11 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/3134809103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:42:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/4240156806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]: dispatch 2024-04-26T22:42:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]: dispatch 2024-04-26T22:42:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa8c529-7d81-4be2-90ee-2ce0cc461c84"}]': finished 2024-04-26T22:42:11.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T22:42:11.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:11.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:11 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/3134809103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:42:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:13 smithi001 ceph-mon[29435]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:13 smithi038 ceph-mon[32433]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:13 smithi186 ceph-mon[32774]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:15.804 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:15 smithi038 ceph-mon[32433]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:15.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:15 smithi001 ceph-mon[29435]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:15.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:15 smithi186 ceph-mon[32774]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:17.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:17 smithi038 ceph-mon[32433]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:17 smithi001 ceph-mon[29435]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:17 smithi186 ceph-mon[32774]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:18.733 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T22:42:18.733 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:18.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T22:42:18.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:18.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T22:42:18.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:19 smithi038 ceph-mon[32433]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:19 smithi038 ceph-mon[32433]: Deploying daemon osd.1 on smithi038 2024-04-26T22:42:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:19 smithi001 ceph-mon[29435]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:19 smithi001 ceph-mon[29435]: Deploying daemon osd.1 on smithi038 2024-04-26T22:42:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:19 smithi186 ceph-mon[32774]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:19 smithi186 ceph-mon[32774]: Deploying daemon osd.1 on smithi038 2024-04-26T22:42:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:21 smithi001 ceph-mon[29435]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:21.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:21 smithi038 ceph-mon[32433]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:21 smithi186 ceph-mon[32774]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:23.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:23 smithi001 ceph-mon[29435]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:23.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:23 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:23.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:23 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:23.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:23 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:23 smithi038 ceph-mon[32433]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:23 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:23 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:23 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:23 smithi186 ceph-mon[32774]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:23 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:23 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:23 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:25.774 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:25 smithi038 ceph-mon[32433]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:25 smithi001 ceph-mon[29435]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:25 smithi186 ceph-mon[32774]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:27.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.339+0000 7fc5957fa700 1 -- 172.21.15.38:0/3095213406 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc5980ffb00 con 0x7fc5800a3bb0 2024-04-26T22:42:27.339 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 1 on host 'smithi038' 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5800a3bb0 msgr2=0x7fc5800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5800a3bb0 0x7fc5800a6070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00ac50 tx=0x7fc58c00a380 comp rx=0 tx=0).stop 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 msgr2=0x7fc598199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc598199570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc58800fa00 tx=0x7fc58800fa30 comp rx=0 tx=0).stop 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 shutdown_connections 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5800a3bb0 0x7fc5800a6070 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc598103f60 0x7fc5981a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5981023c0 0x7fc598199570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:27.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 --2- 172.21.15.38:0/3095213406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5981035c0 0x7fc598199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:27.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.341+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 >> 172.21.15.38:0/3095213406 conn(0x7fc5980fd980 msgr2=0x7fc5980ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:42:27.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.342+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 shutdown_connections 2024-04-26T22:42:27.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:42:27.342+0000 7fc59e7f6700 1 -- 172.21.15.38:0/3095213406 wait complete. 2024-04-26T22:42:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T22:42:27.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:27 smithi038 ceph-mon[32433]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T22:42:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:27.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T22:42:27.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:27 smithi186 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T22:42:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T22:42:27.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:27 smithi001 ceph-mon[29435]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T22:42:28.109 DEBUG:teuthology.orchestra.run.smithi038:osd.1> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T22:42:28.111 INFO:tasks.cephadm:Deploying osd.2 on smithi186 with /dev/vg_nvme/lv_4... 2024-04-26T22:42:28.111 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T22:42:28.357 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:42:29.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:29 smithi186 ceph-mon[32774]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:29.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:29 smithi186 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T22:42:29.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:29 smithi186 ceph-mon[32774]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T22:42:29.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:29.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:29 smithi186 ceph-mon[32774]: from='osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:42:29.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:29 smithi186 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:42:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:29 smithi001 ceph-mon[29435]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:29 smithi001 ceph-mon[29435]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T22:42:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:29 smithi001 ceph-mon[29435]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T22:42:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:29 smithi001 ceph-mon[29435]: from='osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:42:29.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:29 smithi001 ceph-mon[29435]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:42:29.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-mon[32433]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:29.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-mon[32433]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T22:42:29.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-mon[32433]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T22:42:29.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:29.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-mon[32433]: from='osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:42:29.600 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-mon[32433]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:42:29.600 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1[37012]: 2024-04-26T22:42:29.328+0000 7f1e01a73700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T22:42:29.600 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:29 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1[37012]: 2024-04-26T22:42:29.332+0000 7f1dfd866700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:42:30.092 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T22:42:30.093 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T22:42:30.093 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T22:42:30.093 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T22:42:30.093 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115638 s, 907 MB/s 2024-04-26T22:42:30.093 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 2024-04-26T22:42:30.093 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T22:42:30.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:30 smithi186 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:42:30.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:30 smithi186 ceph-mon[32774]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T22:42:30.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:30.393 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:30.595 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-04-26T22:42:30.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:30 smithi001 ceph-mon[29435]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:42:30.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:30 smithi001 ceph-mon[29435]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T22:42:30.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:30.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:30.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:30 smithi038 ceph-mon[32433]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:42:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:30 smithi038 ceph-mon[32433]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T22:42:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:30.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:30.840 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:42:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:42:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:42:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] boot 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:31.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] boot 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:42:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:42:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T22:42:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: osd.1 [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] boot 2024-04-26T22:42:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:31.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 -- 172.21.15.186:0/3179841653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440102380 msgr2=0x7fc4401027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 --2- 172.21.15.186:0/3179841653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440102380 0x7fc4401027a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc430009990 tx=0x7fc430009ca0 comp rx=0 tx=0).stop 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 -- 172.21.15.186:0/3179841653 shutdown_connections 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 --2- 172.21.15.186:0/3179841653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440102380 0x7fc4401027a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 --2- 172.21.15.186:0/3179841653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103f20 0x7fc44010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 --2- 172.21.15.186:0/3179841653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103580 0x7fc4401039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 -- 172.21.15.186:0/3179841653 >> 172.21.15.186:0/3179841653 conn(0x7fc4400fd920 msgr2=0x7fc4400ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 -- 172.21.15.186:0/3179841653 shutdown_connections 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.271+0000 7fc4451a4700 1 -- 172.21.15.186:0/3179841653 wait complete. 2024-04-26T22:42:32.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 Processor -- start 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 -- start start 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440102380 0x7fc440106310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 0x7fc440104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103f20 0x7fc440104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010de10 con 0x7fc440103f20 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010dc90 con 0x7fc440102380 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc4451a4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010df90 con 0x7fc440103580 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 0x7fc440104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 0x7fc440104960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51486/0 (socket says 172.21.15.186:51486) 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 -- 172.21.15.186:0/500452600 learned_addr learned my addr 172.21.15.186:0/500452600 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43ed9d700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440102380 0x7fc440106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43f59e700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103f20 0x7fc440104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:32.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 -- 172.21.15.186:0/500452600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440102380 msgr2=0x7fc440106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440102380 0x7fc440106310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 -- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103f20 msgr2=0x7fc440104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103f20 0x7fc440104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.272+0000 7fc43659c700 1 -- 172.21.15.186:0/500452600 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc440105750 con 0x7fc440103580 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.273+0000 7fc43659c700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 0x7fc440104960 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc42800d980 tx=0x7fc42800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.273+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc428016070 con 0x7fc440103580 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.273+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc428004830 con 0x7fc440103580 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.273+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4401a37b0 con 0x7fc440103580 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.273+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4280054c0 con 0x7fc440103580 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.273+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4401a3cc0 con 0x7fc440103580 2024-04-26T22:42:32.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.274+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc44006a110 con 0x7fc440103580 2024-04-26T22:42:32.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.275+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7fc428004220 con 0x7fc440103580 2024-04-26T22:42:32.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.275+0000 7fc43cd99700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3bb0 0x7fc42c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:32.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.276+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fc4280ceed0 con 0x7fc440103580 2024-04-26T22:42:32.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.276+0000 7fc43ed9d700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3bb0 0x7fc42c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:32.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.276+0000 7fc43ed9d700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3bb0 0x7fc42c0a6070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc43000b5c0 tx=0x7fc430005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:42:32.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.279+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc42801f020 con 0x7fc440103580 2024-04-26T22:42:32.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:32.489+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc4401a3fe0 con 0x7fc42c0a3bb0 2024-04-26T22:42:32.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:32 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi038 2024-04-26T22:42:32.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:32 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-26T22:42:32.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:32 smithi001 ceph-mon[29435]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T22:42:32.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:32 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi038 2024-04-26T22:42:32.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:32 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-26T22:42:32.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:32 smithi038 ceph-mon[32433]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T22:42:32.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:32 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi038 2024-04-26T22:42:32.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:32 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-26T22:42:32.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:32 smithi186 ceph-mon[32774]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T22:42:33.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:33 smithi038 ceph-mon[32433]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:33.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:33 smithi038 ceph-mon[32433]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:33.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:33.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:33.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:33 smithi186 ceph-mon[32774]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:33 smithi186 ceph-mon[32774]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:33.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:33.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:33 smithi001 ceph-mon[29435]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:33 smithi001 ceph-mon[29435]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:35 smithi186 ceph-mon[32774]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:35 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/1634221906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]: dispatch 2024-04-26T22:42:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:35 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]: dispatch 2024-04-26T22:42:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:35 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]': finished 2024-04-26T22:42:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:35 smithi186 ceph-mon[32774]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T22:42:35.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:35 smithi001 ceph-mon[29435]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:35 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/1634221906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]: dispatch 2024-04-26T22:42:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:35 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]: dispatch 2024-04-26T22:42:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:35 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]': finished 2024-04-26T22:42:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:35 smithi001 ceph-mon[29435]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T22:42:35.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:35 smithi038 ceph-mon[32433]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:35 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/1634221906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]: dispatch 2024-04-26T22:42:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:35 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]: dispatch 2024-04-26T22:42:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:35 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d043997-5257-4a78-99a6-4e3483360b0a"}]': finished 2024-04-26T22:42:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:35 smithi038 ceph-mon[32433]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T22:42:35.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:36.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:36 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/80053618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:42:36.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:36 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/80053618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:42:36.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:36 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/80053618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:42:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:37 smithi186 ceph-mon[32774]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:37 smithi001 ceph-mon[29435]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:37 smithi038 ceph-mon[32433]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:39 smithi186 ceph-mon[32774]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:39 smithi001 ceph-mon[29435]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:39 smithi038 ceph-mon[32433]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:41 smithi186 ceph-mon[32774]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:41 smithi001 ceph-mon[29435]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:41 smithi038 ceph-mon[32433]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:42.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T22:42:42.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T22:42:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T22:42:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:43.693 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:43 smithi186 ceph-mon[32774]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:43.694 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:43 smithi186 ceph-mon[32774]: Deploying daemon osd.2 on smithi186 2024-04-26T22:42:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:43 smithi001 ceph-mon[29435]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:43.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:43 smithi001 ceph-mon[29435]: Deploying daemon osd.2 on smithi186 2024-04-26T22:42:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:43 smithi038 ceph-mon[32433]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:43 smithi038 ceph-mon[32433]: Deploying daemon osd.2 on smithi186 2024-04-26T22:42:45.696 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:45 smithi186 ceph-mon[32774]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:45 smithi001 ceph-mon[29435]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:45 smithi038 ceph-mon[32433]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:47.715 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:47 smithi186 ceph-mon[32774]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:47.715 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:47.715 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:47.715 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:47 smithi001 ceph-mon[29435]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:47 smithi038 ceph-mon[32433]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:49 smithi186 ceph-mon[32774]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:49 smithi001 ceph-mon[29435]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:49 smithi038 ceph-mon[32433]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:50.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:50.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.936 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.936 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:50.936 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:50.936 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:50.936 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.006+0000 7fc43cd99700 1 -- 172.21.15.186:0/500452600 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc4401a3fe0 con 0x7fc42c0a3bb0 2024-04-26T22:42:51.008 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 2 on host 'smithi186' 2024-04-26T22:42:51.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3bb0 msgr2=0x7fc42c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3bb0 0x7fc42c0a6070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc43000b5c0 tx=0x7fc430005fb0 comp rx=0 tx=0).stop 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 msgr2=0x7fc440104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 0x7fc440104960 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc42800d980 tx=0x7fc42800dc90 comp rx=0 tx=0).stop 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 shutdown_connections 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3bb0 0x7fc42c0a6070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103f20 0x7fc440104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440102380 0x7fc440106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 --2- 172.21.15.186:0/500452600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440103580 0x7fc440104960 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 >> 172.21.15.186:0/500452600 conn(0x7fc4400fd920 msgr2=0x7fc44006b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 shutdown_connections 2024-04-26T22:42:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:42:51.009+0000 7fc4451a4700 1 -- 172.21.15.186:0/500452600 wait complete. 2024-04-26T22:42:51.625 DEBUG:teuthology.orchestra.run.smithi186:osd.2> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T22:42:51.627 INFO:tasks.cephadm:Deploying osd.3 on smithi001 with /dev/vg_nvme/lv_3... 2024-04-26T22:42:51.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T22:42:51.799 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:51 smithi186 ceph-mon[32774]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:51.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:51.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:51 smithi186 ceph-mon[32774]: from='osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T22:42:51.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:51 smithi186 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T22:42:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:51 smithi001 ceph-mon[29435]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:51 smithi001 ceph-mon[29435]: from='osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T22:42:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:51 smithi001 ceph-mon[29435]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T22:42:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:51 smithi038 ceph-mon[32433]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:51 smithi038 ceph-mon[32433]: from='osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T22:42:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:51 smithi038 ceph-mon[32433]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T22:42:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:42:52.981 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:52 smithi186 ceph-mon[32774]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:52.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:52 smithi186 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T22:42:52.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:52 smithi186 ceph-mon[32774]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T22:42:52.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:52 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:52 smithi186 ceph-mon[32774]: from='osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:42:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:52 smithi186 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:42:53.281 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:52 smithi001 ceph-mon[29435]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:53.281 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:52 smithi001 ceph-mon[29435]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T22:42:53.281 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:52 smithi001 ceph-mon[29435]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T22:42:53.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:52 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:53.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:52 smithi001 ceph-mon[29435]: from='osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:42:53.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:52 smithi001 ceph-mon[29435]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:42:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:52 smithi038 ceph-mon[32433]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:52 smithi038 ceph-mon[32433]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T22:42:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:52 smithi038 ceph-mon[32433]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T22:42:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:52 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:52 smithi038 ceph-mon[32433]: from='osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:42:53.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:52 smithi038 ceph-mon[32433]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:42:53.737 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2[37918]: 2024-04-26T22:42:53.428+0000 7f70f09b4700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T22:42:53.737 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2[37918]: 2024-04-26T22:42:53.432+0000 7f70e879f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:42:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T22:42:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T22:42:53.868 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T22:42:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T22:42:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00866658 s, 1.2 GB/s 2024-04-26T22:42:53.868 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 2024-04-26T22:42:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T22:42:54.119 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:42:54.119 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-mon[32774]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T22:42:54.120 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:54.120 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:54.120 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:53 smithi186 ceph-mon[32774]: from='osd.2 ' entity='osd.2' 2024-04-26T22:42:54.186 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:53 smithi001 ceph-mon[29435]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:42:54.186 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:53 smithi001 ceph-mon[29435]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T22:42:54.186 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:53 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:54.186 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:53 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:54.186 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:53 smithi001 ceph-mon[29435]: from='osd.2 ' entity='osd.2' 2024-04-26T22:42:54.253 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:53 smithi038 ceph-mon[32433]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:42:54.253 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:53 smithi038 ceph-mon[32433]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T22:42:54.253 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:53 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:54.253 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:53 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:54.253 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:53 smithi038 ceph-mon[32433]: from='osd.2 ' entity='osd.2' 2024-04-26T22:42:54.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_3 2024-04-26T22:42:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] boot 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T22:42:55.056 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi186 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:55.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:42:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] boot 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi186 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:55.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:42:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:42:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T22:42:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: osd.2 [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] boot 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi186 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:55.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:56.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:55 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T22:42:56.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 -- 172.21.15.1:0/326307125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 msgr2=0x7fe0a0104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:56.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/326307125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0104ae0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0099c0 tx=0x7fe08c009cd0 comp rx=0 tx=0).stop 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 -- 172.21.15.1:0/326307125 shutdown_connections 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/326307125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 0x7fe0a01038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/326307125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0a0105020 0x7fe0a010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/326307125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0104ae0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 -- 172.21.15.1:0/326307125 >> 172.21.15.1:0/326307125 conn(0x7fe0a00fea20 msgr2=0x7fe0a0100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 -- 172.21.15.1:0/326307125 shutdown_connections 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 -- 172.21.15.1:0/326307125 wait complete. 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 Processor -- start 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.328+0000 7fe0a6d57700 1 -- start start 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 0x7fe0a0199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0a0105020 0x7fe0a01a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010ef70 con 0x7fe0a0104680 2024-04-26T22:42:56.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010edf0 con 0x7fe0a0105020 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010f0f0 con 0x7fe0a0103480 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52544/0 (socket says 172.21.15.1:52544) 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 -- 172.21.15.1:0/114555401 learned_addr learned my addr 172.21.15.1:0/114555401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a52f4700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0a0105020 0x7fe0a01a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a4af3700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 0x7fe0a0199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 -- 172.21.15.1:0/114555401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 msgr2=0x7fe0a0199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 0x7fe0a0199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 -- 172.21.15.1:0/114555401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0a0105020 msgr2=0x7fe0a01a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0a0105020 0x7fe0a01a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 -- 172.21.15.1:0/114555401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a01a12a0 con 0x7fe0a0104680 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a4af3700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 0x7fe0a0199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09ffff700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0199b80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00fa30 tx=0x7fe08c00fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c027070 con 0x7fe0a0104680 2024-04-26T22:42:56.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a01a1530 con 0x7fe0a0104680 2024-04-26T22:42:56.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe08c022920 con 0x7fe0a0104680 2024-04-26T22:42:56.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.329+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c009e60 con 0x7fe0a0104680 2024-04-26T22:42:56.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.330+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a01a19e0 con 0x7fe0a0104680 2024-04-26T22:42:56.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.331+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a006a110 con 0x7fe0a0104680 2024-04-26T22:42:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.332+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7fe08c02a070 con 0x7fe0a0104680 2024-04-26T22:42:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.333+0000 7fe09dffb700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe0900a3bb0 0x7fe0900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:42:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.333+0000 7fe0a4af3700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe0900a3bb0 0x7fe0900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:42:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.333+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fe08c0cffd0 con 0x7fe0a0104680 2024-04-26T22:42:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.333+0000 7fe0a4af3700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe0900a3bb0 0x7fe0900a6070 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe094005950 tx=0x7fe0940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:42:56.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.337+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe08c013070 con 0x7fe0a0104680 2024-04-26T22:42:56.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:55 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T22:42:56.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:55 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T22:42:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:56.553+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe0a0063860 con 0x7fe0900a3bb0 2024-04-26T22:42:57.261 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:42:57.261 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T22:42:57.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T22:42:57.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T22:42:57.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:57.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:57.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:57.262 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:57.320 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46915]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T22:42:57.320 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46915]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.320 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46915]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.320 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46915]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.320 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46918]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T22:42:57.320 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46918]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.321 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46918]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.321 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46918]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:42:57.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T22:42:57.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T22:42:57.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T22:42:57.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:57.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:57.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:57.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:57.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46921]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:42:57.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:57.352 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40737]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T22:42:57.352 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40737]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.353 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40737]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.353 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40737]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.353 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40740]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T22:42:57.353 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40740]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.353 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40740]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.353 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40740]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46921]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46921]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46921]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46925]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46925]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46925]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:57 smithi001 sudo[46925]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41535]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T22:42:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41535]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41535]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T22:42:57.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41538]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41538]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41529]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41532]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41532]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.736 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41532]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.737 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 22:42:57 smithi186 sudo[41532]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40743]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T22:42:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40743]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40743]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40743]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:57.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40746]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T22:42:57.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40746]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T22:42:57.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40746]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T22:42:57.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:57 smithi038 sudo[40746]: pam_unix(sudo:session): session closed for user root 2024-04-26T22:42:58.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:42:58.320+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe08c099e70 con 0x7fe0a0104680 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:58 smithi001 ceph-mon[29435]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T22:42:58.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T22:42:58.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:58 smithi038 ceph-mon[32433]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:58.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T22:42:58.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T22:42:58.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T22:42:58.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T22:42:58.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:58 smithi186 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T22:42:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:59 smithi001 ceph-mon[29435]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:42:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:59 smithi001 ceph-mon[29435]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T22:42:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:42:59 smithi001 ceph-mon[29435]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T22:42:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:59 smithi038 ceph-mon[32433]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:42:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:59 smithi038 ceph-mon[32433]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T22:42:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:42:59 smithi038 ceph-mon[32433]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T22:42:59.388 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:59 smithi186 ceph-mon[32774]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:42:59.388 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:59 smithi186 ceph-mon[32774]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T22:42:59.388 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:42:59 smithi186 ceph-mon[32774]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T22:43:00.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:00 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1630141498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed1bef0e-ba55-4c4a-9423-0024c218b1dc"}]: dispatch 2024-04-26T22:43:00.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:00 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1630141498' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed1bef0e-ba55-4c4a-9423-0024c218b1dc"}]': finished 2024-04-26T22:43:00.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:00 smithi001 ceph-mon[29435]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T22:43:00.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:00 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:00.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:00 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1630141498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed1bef0e-ba55-4c4a-9423-0024c218b1dc"}]: dispatch 2024-04-26T22:43:00.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:00 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1630141498' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed1bef0e-ba55-4c4a-9423-0024c218b1dc"}]': finished 2024-04-26T22:43:00.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:00 smithi038 ceph-mon[32433]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T22:43:00.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:00 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:00.401 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:00 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1630141498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed1bef0e-ba55-4c4a-9423-0024c218b1dc"}]: dispatch 2024-04-26T22:43:00.401 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:00 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1630141498' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed1bef0e-ba55-4c4a-9423-0024c218b1dc"}]': finished 2024-04-26T22:43:00.401 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:00 smithi186 ceph-mon[32774]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T22:43:00.401 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:00 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:01 smithi001 ceph-mon[29435]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:01 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/317622326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:01 smithi038 ceph-mon[32433]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:01 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/317622326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:01 smithi186 ceph-mon[32774]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:01 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/317622326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:03.337 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:03 smithi001 ceph-mon[29435]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:03 smithi038 ceph-mon[32433]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:03.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:03 smithi186 ceph-mon[32774]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:05.209 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:05 smithi001 ceph-mon[29435]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:05.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:05 smithi038 ceph-mon[32433]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:05 smithi186 ceph-mon[32774]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:07.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:07 smithi001 ceph-mon[29435]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:07.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T22:43:07.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:07 smithi038 ceph-mon[32433]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T22:43:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:07 smithi186 ceph-mon[32774]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T22:43:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:08.247 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:08 smithi001 ceph-mon[29435]: Deploying daemon osd.3 on smithi001 2024-04-26T22:43:08.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:08 smithi038 ceph-mon[32433]: Deploying daemon osd.3 on smithi001 2024-04-26T22:43:08.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:08 smithi186 ceph-mon[32774]: Deploying daemon osd.3 on smithi001 2024-04-26T22:43:09.253 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:09 smithi001 ceph-mon[29435]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:09.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:09 smithi038 ceph-mon[32433]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:09 smithi186 ceph-mon[32774]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:11.345 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:11 smithi001 ceph-mon[29435]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:11 smithi038 ceph-mon[32433]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:11 smithi186 ceph-mon[32774]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:13 smithi001 ceph-mon[29435]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:13.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:13.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:13 smithi038 ceph-mon[32433]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:13 smithi186 ceph-mon[32774]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:15 smithi001 ceph-mon[29435]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:15 smithi038 ceph-mon[32433]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:15 smithi186 ceph-mon[32774]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:16.218 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 3 on host 'smithi001' 2024-04-26T22:43:16.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.217+0000 7fe09dffb700 1 -- 172.21.15.1:0/114555401 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe0a0063860 con 0x7fe0900a3bb0 2024-04-26T22:43:16.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.219+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe0900a3bb0 msgr2=0x7fe0900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:16.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.219+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe0900a3bb0 0x7fe0900a6070 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe094005950 tx=0x7fe0940058e0 comp rx=0 tx=0).stop 2024-04-26T22:43:16.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.219+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 msgr2=0x7fe0a0199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:16.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.219+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0199b80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00fa30 tx=0x7fe08c00fa60 comp rx=0 tx=0).stop 2024-04-26T22:43:16.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.219+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 shutdown_connections 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.219+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe0900a3bb0 0x7fe0900a6070 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.220+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0a0103480 0x7fe0a0199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.220+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0a0105020 0x7fe0a01a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.220+0000 7fe0a6d57700 1 --2- 172.21.15.1:0/114555401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a0104680 0x7fe0a0199b80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.220+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 >> 172.21.15.1:0/114555401 conn(0x7fe0a00fea20 msgr2=0x7fe0a0107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.220+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 shutdown_connections 2024-04-26T22:43:16.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:43:16.220+0000 7fe0a6d57700 1 -- 172.21.15.1:0/114555401 wait complete. 2024-04-26T22:43:16.537 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.537 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.537 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:16.537 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:16.537 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.537 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T22:43:16.538 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:16.538 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.538 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:16.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:16.873 DEBUG:teuthology.orchestra.run.smithi001:osd.3> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T22:43:16.876 INFO:tasks.cephadm:Deploying osd.4 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-26T22:43:16.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T22:43:17.131 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:43:17.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:17 smithi038 ceph-mon[32433]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:17.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:17 smithi038 ceph-mon[32433]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T22:43:17.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:17 smithi038 ceph-mon[32433]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T22:43:17.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:17 smithi038 ceph-mon[32433]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:43:17.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:17 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:17 smithi186 ceph-mon[32774]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:17 smithi186 ceph-mon[32774]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T22:43:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:17 smithi186 ceph-mon[32774]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T22:43:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:17 smithi186 ceph-mon[32774]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:43:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:17 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:17.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-mon[29435]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:17.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-mon[29435]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T22:43:17.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-mon[29435]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T22:43:17.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-mon[29435]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:43:17.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:17.795 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T22:43:17.354+0000 7fca6f2c3700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T22:43:17.796 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 22:43:17 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T22:43:17.361+0000 7fca6b0b6700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:43:18.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:18 smithi001 ceph-mon[29435]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:43:18.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:18 smithi001 ceph-mon[29435]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T22:43:18.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.592 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:18 smithi038 ceph-mon[32433]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:43:18.592 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:18 smithi038 ceph-mon[32433]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T22:43:18.592 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.592 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.592 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:18 smithi186 ceph-mon[32774]: from='osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:43:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:18 smithi186 ceph-mon[32774]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T22:43:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00927394 s, 1.1 GB/s 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 2024-04-26T22:43:19.159 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T22:43:19.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:19 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:43:19.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:19 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:43:19.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:19 smithi038 ceph-mon[32433]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:19.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:19 smithi038 ceph-mon[32433]: osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] boot 2024-04-26T22:43:19.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:19 smithi038 ceph-mon[32433]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T22:43:19.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:19 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:43:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:19 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:43:19.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:19 smithi001 ceph-mon[29435]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:19.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:19 smithi001 ceph-mon[29435]: osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] boot 2024-04-26T22:43:19.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:19 smithi001 ceph-mon[29435]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T22:43:19.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:19 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:43:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:19 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:43:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:19 smithi186 ceph-mon[32774]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T22:43:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:19 smithi186 ceph-mon[32774]: osd.3 [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] boot 2024-04-26T22:43:19.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:19 smithi186 ceph-mon[32774]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T22:43:19.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T22:43:19.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-26T22:43:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:20.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T22:43:20.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:20.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:20.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:20.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:20.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T22:43:20.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:20.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:20.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:20.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:20.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:20.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:21.509 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:21 smithi038 ceph-mon[32433]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:21.509 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:21 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi001 2024-04-26T22:43:21.509 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:21 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi001 to 8546M 2024-04-26T22:43:21.509 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:21 smithi038 ceph-mon[32433]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T22:43:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:21 smithi186 ceph-mon[32774]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:21 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi001 2024-04-26T22:43:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:21 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi001 to 8546M 2024-04-26T22:43:21.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:21 smithi186 ceph-mon[32774]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T22:43:21.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:21 smithi001 ceph-mon[29435]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:21.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:21 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi001 2024-04-26T22:43:21.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:21 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi001 to 8546M 2024-04-26T22:43:21.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:21 smithi001 ceph-mon[29435]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 -- 172.21.15.38:0/1098651274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc102380 msgr2=0x7f4fcc1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/1098651274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc102380 0x7f4fcc1027a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80099c0 tx=0x7f4fc8009cd0 comp rx=0 tx=0).stop 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 -- 172.21.15.38:0/1098651274 shutdown_connections 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/1098651274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc103f20 0x7f4fcc10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/1098651274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc102380 0x7f4fcc1027a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/1098651274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fcc103580 0x7f4fcc1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 -- 172.21.15.38:0/1098651274 >> 172.21.15.38:0/1098651274 conn(0x7f4fcc0fd920 msgr2=0x7f4fcc0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 -- 172.21.15.38:0/1098651274 shutdown_connections 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.716+0000 7f4fd3f91700 1 -- 172.21.15.38:0/1098651274 wait complete. 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 Processor -- start 2024-04-26T22:43:21.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 -- start start 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc102380 0x7f4fcc06bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 0x7f4fcc19e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fcc103f20 0x7f4fcc1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc10de60 con 0x7f4fcc102380 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc10dce0 con 0x7f4fcc103580 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd3f91700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc10dfe0 con 0x7f4fcc103f20 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 0x7f4fcc19e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 0x7f4fcc19e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55718/0 (socket says 172.21.15.38:55718) 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 -- 172.21.15.38:0/4148373094 learned_addr learned my addr 172.21.15.38:0/4148373094 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd1d2d700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc102380 0x7f4fcc06bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd252e700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fcc103f20 0x7f4fcc1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 -- 172.21.15.38:0/4148373094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fcc103f20 msgr2=0x7f4fcc1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:21.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fcc103f20 0x7f4fcc1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 -- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc102380 msgr2=0x7f4fcc06bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc102380 0x7f4fcc06bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd152c700 1 -- 172.21.15.38:0/4148373094 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc1a1280 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.717+0000 7f4fd1d2d700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc102380 0x7f4fcc06bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.718+0000 7f4fd152c700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 0x7f4fcc19e370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00c970 tx=0x7f4fbc00cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.718+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc007aa0 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.718+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fbc004ce0 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.718+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fcc1a1570 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.718+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc0055c0 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.718+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fcc1a1ab0 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.719+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fcc06a110 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.720+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4fbc007c40 con 0x7f4fcc103580 2024-04-26T22:43:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.721+0000 7f4fc2ffd700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4fb80a3cc0 0x7f4fb80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:21.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.721+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f4fbc0cf230 con 0x7f4fcc103580 2024-04-26T22:43:21.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.721+0000 7f4fd1d2d700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4fb80a3cc0 0x7f4fb80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:21.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.722+0000 7f4fd1d2d700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4fb80a3cc0 0x7f4fb80a6180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8000c00 tx=0x7f4fc8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:43:21.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.725+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4fbc09cf50 con 0x7f4fcc103580 2024-04-26T22:43:21.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:21.943+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4fcc063860 con 0x7f4fb80a3cc0 2024-04-26T22:43:22.701 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:22 smithi038 ceph-mon[32433]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T22:43:22.701 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:43:22.701 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:43:22.701 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:22 smithi186 ceph-mon[32774]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T22:43:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:43:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:43:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:22.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:22 smithi001 ceph-mon[29435]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T22:43:22.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:43:22.764 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:43:22.764 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:23.674 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:23 smithi038 ceph-mon[32433]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T22:43:23.674 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:23 smithi038 ceph-mon[32433]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:43:23.674 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:23 smithi038 ceph-mon[32433]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T22:43:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:23 smithi186 ceph-mon[32774]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T22:43:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:23 smithi186 ceph-mon[32774]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:43:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:23 smithi186 ceph-mon[32774]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T22:43:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:23 smithi001 ceph-mon[29435]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T22:43:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:23 smithi001 ceph-mon[29435]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:43:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:23 smithi001 ceph-mon[29435]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T22:43:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-26T22:43:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/430404216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]: dispatch 2024-04-26T22:43:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]: dispatch 2024-04-26T22:43:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]': finished 2024-04-26T22:43:25.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T22:43:25.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:25.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:25 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/1505596237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-26T22:43:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/430404216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]: dispatch 2024-04-26T22:43:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]: dispatch 2024-04-26T22:43:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]': finished 2024-04-26T22:43:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T22:43:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:25 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/1505596237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/430404216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]: dispatch 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]: dispatch 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98421e8d-87a9-4a37-9415-d806e7d31651"}]': finished 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:25.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:25 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/1505596237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:27 smithi186 ceph-mon[32774]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-26T22:43:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:27 smithi001 ceph-mon[29435]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-26T22:43:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:27 smithi038 ceph-mon[32433]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-26T22:43:29.673 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:29 smithi038 ceph-mon[32433]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:29 smithi186 ceph-mon[32774]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:29 smithi001 ceph-mon[29435]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:31.710 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:31 smithi038 ceph-mon[32433]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:31 smithi186 ceph-mon[32774]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:31 smithi001 ceph-mon[29435]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:32.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T22:43:32.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:32.761 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T22:43:32.761 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T22:43:32.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:33.692 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:33 smithi038 ceph-mon[32433]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:33.692 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:33 smithi038 ceph-mon[32433]: Deploying daemon osd.4 on smithi038 2024-04-26T22:43:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:33 smithi186 ceph-mon[32774]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:33 smithi186 ceph-mon[32774]: Deploying daemon osd.4 on smithi038 2024-04-26T22:43:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:33 smithi001 ceph-mon[29435]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:33 smithi001 ceph-mon[29435]: Deploying daemon osd.4 on smithi038 2024-04-26T22:43:35.686 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:35 smithi038 ceph-mon[32433]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:35 smithi186 ceph-mon[32774]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:35 smithi001 ceph-mon[29435]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:37 smithi186 ceph-mon[32774]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:37 smithi001 ceph-mon[29435]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:37 smithi038 ceph-mon[32433]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:38.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:38.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:38.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:38.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:38 smithi186 ceph-mon[32774]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:39.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:39.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:39.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:39.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:38 smithi001 ceph-mon[29435]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:39.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:39.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:39.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:39.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:38 smithi038 ceph-mon[32433]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:41.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.361+0000 7f4fc2ffd700 1 -- 172.21.15.38:0/4148373094 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4fcc063860 con 0x7f4fb80a3cc0 2024-04-26T22:43:41.363 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 4 on host 'smithi038' 2024-04-26T22:43:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4fb80a3cc0 msgr2=0x7f4fb80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4fb80a3cc0 0x7f4fb80a6180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8000c00 tx=0x7f4fc8005fb0 comp rx=0 tx=0).stop 2024-04-26T22:43:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 msgr2=0x7f4fcc19e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 0x7f4fcc19e370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00c970 tx=0x7f4fbc00cd30 comp rx=0 tx=0).stop 2024-04-26T22:43:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 shutdown_connections 2024-04-26T22:43:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4fb80a3cc0 0x7f4fb80a6180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fcc103f20 0x7f4fcc1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fcc102380 0x7f4fcc06bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 --2- 172.21.15.38:0/4148373094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fcc103580 0x7f4fcc19e370 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.363+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 >> 172.21.15.38:0/4148373094 conn(0x7f4fcc0fd920 msgr2=0x7f4fcc1069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:43:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.364+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 shutdown_connections 2024-04-26T22:43:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:43:41.364+0000 7f4fd3f91700 1 -- 172.21.15.38:0/4148373094 wait complete. 2024-04-26T22:43:41.570 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:41.570 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.570 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.570 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:41.570 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:41.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.571 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:41.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:41.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:42.022 DEBUG:teuthology.orchestra.run.smithi038:osd.4> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T22:43:42.024 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-04-26T22:43:42.024 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T22:43:42.272 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:43:42.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:42 smithi038 ceph-mon[32433]: from='osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T22:43:42.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:42 smithi038 ceph-mon[32433]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T22:43:42.672 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:42 smithi186 ceph-mon[32774]: from='osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T22:43:42.672 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:42 smithi186 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T22:43:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:42 smithi001 ceph-mon[29435]: from='osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T22:43:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:42 smithi001 ceph-mon[29435]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T22:43:43.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:43 smithi186 ceph-mon[32774]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:43.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:43 smithi186 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T22:43:43.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:43 smithi186 ceph-mon[32774]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T22:43:43.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:43 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:43.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:43 smithi186 ceph-mon[32774]: from='osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:43:43.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:43 smithi186 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:43:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:43 smithi001 ceph-mon[29435]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:43 smithi001 ceph-mon[29435]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T22:43:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:43 smithi001 ceph-mon[29435]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T22:43:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:43 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:43.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:43 smithi001 ceph-mon[29435]: from='osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:43:43.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:43 smithi001 ceph-mon[29435]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:43:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-mon[32433]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-mon[32433]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T22:43:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-mon[32433]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T22:43:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-mon[32433]: from='osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:43:43.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-mon[32433]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:43:43.852 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4[43545]: 2024-04-26T22:43:43.449+0000 7ff7c132d700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T22:43:43.852 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 22:43:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4[43545]: 2024-04-26T22:43:43.456+0000 7ff7bd120700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:43:44.223 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T22:43:44.223 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T22:43:44.223 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T22:43:44.224 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T22:43:44.224 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117379 s, 893 MB/s 2024-04-26T22:43:44.224 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T22:43:44.525 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:44 smithi186 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:43:44.526 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:44 smithi186 ceph-mon[32774]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T22:43:44.526 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:44.526 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:44 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:44.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:44 smithi038 ceph-mon[32433]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:43:44.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:44 smithi038 ceph-mon[32433]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T22:43:44.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:44.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:44 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:44.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:44 smithi001 ceph-mon[29435]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:43:44.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:44 smithi001 ceph-mon[29435]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T22:43:44.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:44.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:44 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:44.952 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-04-26T22:43:45.195 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:43:45.661 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] boot 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:45.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:43:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] boot 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:45.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:43:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:43:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T22:43:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: osd.4 [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] boot 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:45.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:43:46.646 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:46 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi038 2024-04-26T22:43:46.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:46 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-26T22:43:46.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:46 smithi186 ceph-mon[32774]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T22:43:46.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:46 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi038 2024-04-26T22:43:46.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:46 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-26T22:43:46.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:46 smithi001 ceph-mon[29435]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T22:43:46.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 -- 172.21.15.186:0/3786811339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0105110 msgr2=0x7f09e010b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:46.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 --2- 172.21.15.186:0/3786811339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0105110 0x7f09e010b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f09d800b0a0 tx=0x7f09d800b3b0 comp rx=0 tx=0).stop 2024-04-26T22:43:46.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 -- 172.21.15.186:0/3786811339 shutdown_connections 2024-04-26T22:43:46.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 --2- 172.21.15.186:0/3786811339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0105110 0x7f09e010b9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:46.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 --2- 172.21.15.186:0/3786811339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e0101790 0x7f09e0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:46.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 --2- 172.21.15.186:0/3786811339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09e0104770 0x7f09e0104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 -- 172.21.15.186:0/3786811339 >> 172.21.15.186:0/3786811339 conn(0x7f09e00fd0f0 msgr2=0x7f09e00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 -- 172.21.15.186:0/3786811339 shutdown_connections 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 -- 172.21.15.186:0/3786811339 wait complete. 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 Processor -- start 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.848+0000 7f09e68ba700 1 -- start start 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e68ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09e0101790 0x7f09e019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e68ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 0x7f09e019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e68ba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e0105110 0x7f09e01a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e68ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09e010efb0 con 0x7f09e0101790 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e68ba700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09e010ee30 con 0x7f09e0105110 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e68ba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09e010f130 con 0x7f09e0104770 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 0x7f09e019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:46.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 0x7f09e019bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39470/0 (socket says 172.21.15.186:39470) 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 -- 172.21.15.186:0/1829779808 learned_addr learned my addr 172.21.15.186:0/1829779808 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09e4e57700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e0105110 0x7f09e01a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09dffff700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09e0101790 0x7f09e019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 -- 172.21.15.186:0/1829779808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e0105110 msgr2=0x7f09e01a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e0105110 0x7f09e01a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 -- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09e0101790 msgr2=0x7f09e019b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09e0101790 0x7f09e019b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 -- 172.21.15.186:0/1829779808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e01a33e0 con 0x7f09e0104770 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.849+0000 7f09df7fe700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 0x7f09e019bcc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f09d400e9d0 tx=0x7f09d400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.850+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d400cbd0 con 0x7f09e0104770 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.850+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09d400cd70 con 0x7f09e0104770 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.850+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e01a36d0 con 0x7f09e0104770 2024-04-26T22:43:46.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.850+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4018680 con 0x7f09e0104770 2024-04-26T22:43:46.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.850+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e01a3c10 con 0x7f09e0104770 2024-04-26T22:43:46.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.851+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e006a110 con 0x7f09e0104770 2024-04-26T22:43:46.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:46 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi038 2024-04-26T22:43:46.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:46 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-26T22:43:46.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:46 smithi038 ceph-mon[32433]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T22:43:46.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.852+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f09d4010430 con 0x7f09e0104770 2024-04-26T22:43:46.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.852+0000 7f09dd7fa700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f09c80a3c40 0x7f09c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:43:46.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.852+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f09d40cf5e0 con 0x7f09e0104770 2024-04-26T22:43:46.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.852+0000 7f09dffff700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f09c80a3c40 0x7f09c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:43:46.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.853+0000 7f09dffff700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f09c80a3c40 0x7f09c80a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09d0005950 tx=0x7f09d00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:43:46.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:46.857+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f09d409d030 con 0x7f09e0104770 2024-04-26T22:43:47.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:43:47.065+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f09e0103c80 con 0x7f09c80a3c40 2024-04-26T22:43:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:47 smithi186 ceph-mon[32774]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:43:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:43:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:47 smithi001 ceph-mon[29435]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:43:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:43:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:47 smithi038 ceph-mon[32433]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:43:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:43:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:48.629 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:48 smithi186 ceph-mon[32774]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:43:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:48 smithi001 ceph-mon[29435]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:43:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:48 smithi038 ceph-mon[32433]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:43:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:49 smithi186 ceph-mon[32774]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:49 smithi001 ceph-mon[29435]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:49 smithi038 ceph-mon[32433]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:50 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/3594420735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]: dispatch 2024-04-26T22:43:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:50 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]: dispatch 2024-04-26T22:43:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:50 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]': finished 2024-04-26T22:43:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:50 smithi186 ceph-mon[32774]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T22:43:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:43:50.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:50 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/70758687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:50 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/3594420735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]: dispatch 2024-04-26T22:43:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:50 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]: dispatch 2024-04-26T22:43:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:50 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]': finished 2024-04-26T22:43:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:50 smithi001 ceph-mon[29435]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T22:43:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:43:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:50 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/70758687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:50 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/3594420735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]: dispatch 2024-04-26T22:43:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:50 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]: dispatch 2024-04-26T22:43:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:50 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6b59471-baf0-4b9b-8756-43142ee043fe"}]': finished 2024-04-26T22:43:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:50 smithi038 ceph-mon[32433]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T22:43:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:43:50.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:50 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/70758687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:43:51.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:51 smithi186 ceph-mon[32774]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:51.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:51 smithi001 ceph-mon[29435]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:51 smithi038 ceph-mon[32433]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:53 smithi186 ceph-mon[32774]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:53 smithi001 ceph-mon[29435]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:53 smithi038 ceph-mon[32433]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:55 smithi186 ceph-mon[32774]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:55 smithi001 ceph-mon[29435]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:55 smithi038 ceph-mon[32433]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:57.765 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:57 smithi186 ceph-mon[32774]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:57.765 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T22:43:57.765 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:57 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:57 smithi001 ceph-mon[29435]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T22:43:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:57 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:57 smithi038 ceph-mon[32433]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T22:43:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:57 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:58.757 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:58 smithi186 ceph-mon[32774]: Deploying daemon osd.5 on smithi186 2024-04-26T22:43:58.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:58 smithi001 ceph-mon[29435]: Deploying daemon osd.5 on smithi186 2024-04-26T22:43:58.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:58 smithi038 ceph-mon[32433]: Deploying daemon osd.5 on smithi186 2024-04-26T22:43:59.563 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:43:59 smithi186 ceph-mon[32774]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:43:59 smithi001 ceph-mon[29435]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:43:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:43:59 smithi038 ceph-mon[32433]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:01.604 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:01 smithi186 ceph-mon[32774]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:01 smithi001 ceph-mon[29435]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:01 smithi038 ceph-mon[32433]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:03 smithi001 ceph-mon[29435]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:03 smithi038 ceph-mon[32433]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:03 smithi186 ceph-mon[32774]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:03.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:03.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:03.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:05 smithi186 ceph-mon[32774]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:05.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:05.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:05.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:05 smithi001 ceph-mon[29435]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:05 smithi038 ceph-mon[32433]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:05.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:05.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:06.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.287+0000 7f09dd7fa700 1 -- 172.21.15.186:0/1829779808 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f09e0103c80 con 0x7f09c80a3c40 2024-04-26T22:44:06.287 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f09c80a3c40 msgr2=0x7f09c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f09c80a3c40 0x7f09c80a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09d0005950 tx=0x7f09d00058e0 comp rx=0 tx=0).stop 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 msgr2=0x7f09e019bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 0x7f09e019bcc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f09d400e9d0 tx=0x7f09d400ece0 comp rx=0 tx=0).stop 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 shutdown_connections 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f09c80a3c40 0x7f09c80a6100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e0105110 0x7f09e01a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09e0101790 0x7f09e019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 --2- 172.21.15.186:0/1829779808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e0104770 0x7f09e019bcc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:06.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 >> 172.21.15.186:0/1829779808 conn(0x7f09e00fd0f0 msgr2=0x7f09e0102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:44:06.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 shutdown_connections 2024-04-26T22:44:06.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:44:06.289+0000 7f09e68ba700 1 -- 172.21.15.186:0/1829779808 wait complete. 2024-04-26T22:44:06.931 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T22:44:06.933 INFO:tasks.cephadm:Deploying osd.6 on smithi001 with /dev/vg_nvme/lv_2... 2024-04-26T22:44:06.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T22:44:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:44:07.562 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:07 smithi001 ceph-mon[29435]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:07.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:07.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:07.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:07.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:07 smithi001 ceph-mon[29435]: from='osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T22:44:07.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:07 smithi001 ceph-mon[29435]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T22:44:07.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:07 smithi038 ceph-mon[32433]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:07.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:07.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:07.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:07.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:07 smithi038 ceph-mon[32433]: from='osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T22:44:07.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:07 smithi038 ceph-mon[32433]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T22:44:07.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:07 smithi186 ceph-mon[32774]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:07.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:07.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:07.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:07.720 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:07 smithi186 ceph-mon[32774]: from='osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T22:44:07.721 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:07 smithi186 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T22:44:08.550 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:08 smithi001 ceph-mon[29435]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T22:44:08.551 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:08 smithi001 ceph-mon[29435]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T22:44:08.551 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:08.551 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:08 smithi001 ceph-mon[29435]: from='osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:44:08.551 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:08 smithi001 ceph-mon[29435]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:44:08.576 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T22:44:08.576 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-mon[32774]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T22:44:08.576 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:08.576 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-mon[32774]: from='osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:44:08.576 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:44:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:08 smithi038 ceph-mon[32433]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T22:44:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:08 smithi038 ceph-mon[32433]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T22:44:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:08 smithi038 ceph-mon[32433]: from='osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:44:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:08 smithi038 ceph-mon[32433]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:44:08.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T22:44:08.689+0000 7f6d6d2b5700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T22:44:08.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 22:44:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T22:44:08.696+0000 7f6d6489f700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:44:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T22:44:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T22:44:09.256 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T22:44:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T22:44:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00879681 s, 1.2 GB/s 2024-04-26T22:44:09.256 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T22:44:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:44:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T22:44:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: from='osd.5 ' entity='osd.5' 2024-04-26T22:44:09.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:44:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T22:44:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: from='osd.5 ' entity='osd.5' 2024-04-26T22:44:09.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:09.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:44:09.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T22:44:09.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: from='osd.5 ' entity='osd.5' 2024-04-26T22:44:09.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:09.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_2 2024-04-26T22:44:10.238 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] boot 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:10.581 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.582 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:10.582 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:10.582 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:44:10.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:44:10.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] boot 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:10.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:10.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:44:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:44:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: osd.5 [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] boot 2024-04-26T22:44:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:10.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:11 smithi001 ceph-mon[29435]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:11 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi186 2024-04-26T22:44:11.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:11 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T22:44:11.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:11 smithi001 ceph-mon[29435]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T22:44:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:11 smithi038 ceph-mon[32433]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:11 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi186 2024-04-26T22:44:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:11 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T22:44:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:11 smithi038 ceph-mon[32433]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T22:44:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:11 smithi186 ceph-mon[32774]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T22:44:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:11 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi186 2024-04-26T22:44:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:11 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T22:44:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:11 smithi186 ceph-mon[32774]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 -- 172.21.15.1:0/1620952620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c102380 msgr2=0x7fc14c1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1620952620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c102380 0x7fc14c1027a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0099c0 tx=0x7fc13c009cd0 comp rx=0 tx=0).stop 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 -- 172.21.15.1:0/1620952620 shutdown_connections 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1620952620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc14c103f20 0x7fc14c10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1620952620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c102380 0x7fc14c1027a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1620952620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14c103580 0x7fc14c1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 -- 172.21.15.1:0/1620952620 >> 172.21.15.1:0/1620952620 conn(0x7fc14c0fd920 msgr2=0x7fc14c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 -- 172.21.15.1:0/1620952620 shutdown_connections 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.877+0000 7fc1529fe700 1 -- 172.21.15.1:0/1620952620 wait complete. 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 Processor -- start 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 -- start start 2024-04-26T22:44:11.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14c102380 0x7fc14c06bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 0x7fc14c19e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc14c103f20 0x7fc14c1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c10de70 con 0x7fc14c103580 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c10dcf0 con 0x7fc14c103f20 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc1529fe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c10dff0 con 0x7fc14c102380 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc14b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 0x7fc14c19e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc14b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 0x7fc14c19e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53862/0 (socket says 172.21.15.1:53862) 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.878+0000 7fc14b7fe700 1 -- 172.21.15.1:0/1719383829 learned_addr learned my addr 172.21.15.1:0/1719383829 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc150f9b700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc14c103f20 0x7fc14c1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14bfff700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14c102380 0x7fc14c06bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14b7fe700 1 -- 172.21.15.1:0/1719383829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14c102380 msgr2=0x7fc14c06bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14b7fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14c102380 0x7fc14c06bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14b7fe700 1 -- 172.21.15.1:0/1719383829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc14c103f20 msgr2=0x7fc14c1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14b7fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc14c103f20 0x7fc14c1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14b7fe700 1 -- 172.21.15.1:0/1719383829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14c1a12b0 con 0x7fc14c103580 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc14b7fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 0x7fc14c19e3a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc14000b530 tx=0x7fc14000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14001d070 con 0x7fc14c103580 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc140010e20 con 0x7fc14c103580 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14c1981d0 con 0x7fc14c103580 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140016380 con 0x7fc14c103580 2024-04-26T22:44:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.879+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14c198710 con 0x7fc14c103580 2024-04-26T22:44:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.880+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14c06a110 con 0x7fc14c103580 2024-04-26T22:44:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.881+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc140020070 con 0x7fc14c103580 2024-04-26T22:44:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.882+0000 7fc1497fa700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc1340a3c40 0x7fc1340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.882+0000 7fc14bfff700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc1340a3c40 0x7fc1340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.882+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7fc1400d2680 con 0x7fc14c103580 2024-04-26T22:44:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.883+0000 7fc14bfff700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc1340a3c40 0x7fc1340a6100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:44:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:11.887+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc14009ff20 con 0x7fc14c103580 2024-04-26T22:44:12.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:12.106+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc14c063860 con 0x7fc1340a3c40 2024-04-26T22:44:12.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:12 smithi001 ceph-mon[29435]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T22:44:12.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:12 smithi001 ceph-mon[29435]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T22:44:12.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:44:12.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:44:12.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:12.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:12 smithi038 ceph-mon[32433]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T22:44:12.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:12 smithi038 ceph-mon[32433]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T22:44:12.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:44:12.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:44:12.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:12.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:12 smithi186 ceph-mon[32774]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T22:44:12.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:12 smithi186 ceph-mon[32774]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T22:44:12.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:44:12.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:44:12.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:13 smithi001 ceph-mon[29435]: pgmap v141: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:13 smithi001 ceph-mon[29435]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:44:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:13 smithi001 ceph-mon[29435]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T22:44:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:13 smithi001 ceph-mon[29435]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T22:44:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:13 smithi038 ceph-mon[32433]: pgmap v141: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:13 smithi038 ceph-mon[32433]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:44:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:13 smithi038 ceph-mon[32433]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T22:44:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:13 smithi038 ceph-mon[32433]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T22:44:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:13 smithi186 ceph-mon[32774]: pgmap v141: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:13 smithi186 ceph-mon[32774]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:44:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:13 smithi186 ceph-mon[32774]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T22:44:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:13 smithi186 ceph-mon[32774]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T22:44:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:15 smithi001 ceph-mon[29435]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:15.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:15 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3448182350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc08c46f-8ce2-448c-a9fb-74b7559b0d19"}]: dispatch 2024-04-26T22:44:15.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:15 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3448182350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc08c46f-8ce2-448c-a9fb-74b7559b0d19"}]': finished 2024-04-26T22:44:15.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:15 smithi001 ceph-mon[29435]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T22:44:15.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:15 smithi038 ceph-mon[32433]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:15 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3448182350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc08c46f-8ce2-448c-a9fb-74b7559b0d19"}]: dispatch 2024-04-26T22:44:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:15 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3448182350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc08c46f-8ce2-448c-a9fb-74b7559b0d19"}]': finished 2024-04-26T22:44:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:15 smithi038 ceph-mon[32433]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T22:44:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:15 smithi186 ceph-mon[32774]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:15 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3448182350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc08c46f-8ce2-448c-a9fb-74b7559b0d19"}]: dispatch 2024-04-26T22:44:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:15 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3448182350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc08c46f-8ce2-448c-a9fb-74b7559b0d19"}]': finished 2024-04-26T22:44:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:15 smithi186 ceph-mon[32774]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T22:44:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:16.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:16 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3638337355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:44:16.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:16 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3638337355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:44:16.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:16 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3638337355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:44:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:17 smithi001 ceph-mon[29435]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:17.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:17 smithi038 ceph-mon[32433]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:17 smithi186 ceph-mon[32774]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:44:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:18 smithi186 ceph-mon[32774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T22:44:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:18 smithi186 ceph-mon[32774]: Cluster is now healthy 2024-04-26T22:44:18.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:18 smithi001 ceph-mon[29435]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T22:44:18.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:18 smithi001 ceph-mon[29435]: Cluster is now healthy 2024-04-26T22:44:18.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:18 smithi038 ceph-mon[32433]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T22:44:18.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:18 smithi038 ceph-mon[32433]: Cluster is now healthy 2024-04-26T22:44:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:19 smithi186 ceph-mon[32774]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T22:44:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:19 smithi001 ceph-mon[29435]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T22:44:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:19 smithi038 ceph-mon[32433]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T22:44:21.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:21 smithi001 ceph-mon[29435]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:44:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:21 smithi186 ceph-mon[32774]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:44:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:21 smithi038 ceph-mon[32433]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:44:23.684 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:23 smithi001 ceph-mon[29435]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T22:44:23.684 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:23 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T22:44:23.684 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:23 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:23 smithi186 ceph-mon[32774]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T22:44:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:23 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T22:44:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:23 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:23 smithi038 ceph-mon[32433]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T22:44:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:23 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T22:44:23.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:23 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:24.685 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:24 smithi001 ceph-mon[29435]: Deploying daemon osd.6 on smithi001 2024-04-26T22:44:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:24 smithi186 ceph-mon[32774]: Deploying daemon osd.6 on smithi001 2024-04-26T22:44:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:24 smithi038 ceph-mon[32433]: Deploying daemon osd.6 on smithi001 2024-04-26T22:44:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:25 smithi186 ceph-mon[32774]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:44:25.752 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:25 smithi038 ceph-mon[32433]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:44:25.838 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:25 smithi001 ceph-mon[29435]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:44:27.714 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:27 smithi001 ceph-mon[29435]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-26T22:44:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:27 smithi186 ceph-mon[32774]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-26T22:44:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:27 smithi038 ceph-mon[32433]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-26T22:44:29.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:29 smithi001 ceph-mon[29435]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T22:44:29.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:29.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:29.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:29 smithi186 ceph-mon[32774]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T22:44:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:29 smithi038 ceph-mon[32433]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T22:44:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:29.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:31 smithi186 ceph-mon[32774]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:31.757 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:31 smithi001 ceph-mon[29435]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:31 smithi038 ceph-mon[32433]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:33.022 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 6 on host 'smithi001' 2024-04-26T22:44:33.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.021+0000 7fc1497fa700 1 -- 172.21.15.1:0/1719383829 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc14c063860 con 0x7fc1340a3c40 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc1340a3c40 msgr2=0x7fc1340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc1340a3c40 0x7fc1340a6100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c005fb0 comp rx=0 tx=0).stop 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 msgr2=0x7fc14c19e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 0x7fc14c19e3a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc14000b530 tx=0x7fc14000b840 comp rx=0 tx=0).stop 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 shutdown_connections 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc1340a3c40 0x7fc1340a6100 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14c102380 0x7fc14c06bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc14c103f20 0x7fc14c1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 --2- 172.21.15.1:0/1719383829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc14c103580 0x7fc14c19e3a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 >> 172.21.15.1:0/1719383829 conn(0x7fc14c0fd920 msgr2=0x7fc14c1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 shutdown_connections 2024-04-26T22:44:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:44:33.024+0000 7fc1529fe700 1 -- 172.21.15.1:0/1719383829 wait complete. 2024-04-26T22:44:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:33.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:33.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:32 smithi186 ceph-mon[32774]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:32 smithi001 ceph-mon[29435]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:32 smithi038 ceph-mon[32433]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T22:44:33.727 DEBUG:teuthology.orchestra.run.smithi001:osd.6> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T22:44:33.729 INFO:tasks.cephadm:Deploying osd.7 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-26T22:44:33.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T22:44:33.985 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:44:34.269 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:34.269 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:34.269 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:34.270 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T22:44:34.270 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T22:44:34.270 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:44:34.270 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:34.272 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:34.272 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:34.272 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:34.272 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T22:44:34.272 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T22:44:34.273 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:44:34.273 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:34.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T22:44:34.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T22:44:34.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:44:34.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.241 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T22:44:35.028+0000 7f088d7f6700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T22:44:35.241 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T22:44:35.040+0000 7f08855e1700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:44:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-mon[29435]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:35.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-mon[29435]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:44:35.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-mon[29435]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T22:44:35.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:35 smithi001 ceph-mon[29435]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' 2024-04-26T22:44:35.708 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:35 smithi038 ceph-mon[32433]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:35.708 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:35 smithi038 ceph-mon[32433]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:44:35.709 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:35 smithi038 ceph-mon[32433]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T22:44:35.709 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.709 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.709 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:35 smithi038 ceph-mon[32433]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' 2024-04-26T22:44:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:35 smithi186 ceph-mon[32774]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:35 smithi186 ceph-mon[32774]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:44:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:35 smithi186 ceph-mon[32774]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T22:44:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:35 smithi186 ceph-mon[32774]: from='osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713]' entity='osd.6' 2024-04-26T22:44:36.097 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T22:44:36.097 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T22:44:36.097 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T22:44:36.097 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T22:44:36.097 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0155082 s, 676 MB/s 2024-04-26T22:44:36.097 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T22:44:36.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:36 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:44:36.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:36 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:44:36.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:36.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:36 smithi186 ceph-mon[32774]: osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] boot 2024-04-26T22:44:36.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:36 smithi186 ceph-mon[32774]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T22:44:36.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:36.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:36 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:44:36.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:36 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:44:36.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:36.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:36 smithi038 ceph-mon[32433]: osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] boot 2024-04-26T22:44:36.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:36 smithi038 ceph-mon[32433]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T22:44:36.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:36.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-26T22:44:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:36 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:44:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:36 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:44:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:36 smithi001 ceph-mon[29435]: osd.6 [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] boot 2024-04-26T22:44:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:36 smithi001 ceph-mon[29435]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T22:44:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T22:44:37.038 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:44:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:37.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:37 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:37.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:37.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:37 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:37.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:37.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:37 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:38.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 -- 172.21.15.38:0/1117170160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8103f10 msgr2=0x7fbca810a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:38.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1117170160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8103f10 0x7fbca810a7c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b0d0 tx=0x7fbca000b3e0 comp rx=0 tx=0).stop 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 -- 172.21.15.38:0/1117170160 shutdown_connections 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1117170160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8103f10 0x7fbca810a7c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1117170160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8102370 0x7fbca8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1117170160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca8103570 0x7fbca81039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 -- 172.21.15.38:0/1117170160 >> 172.21.15.38:0/1117170160 conn(0x7fbca80fd910 msgr2=0x7fbca80ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 -- 172.21.15.38:0/1117170160 shutdown_connections 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.635+0000 7fbcae36f700 1 -- 172.21.15.38:0/1117170160 wait complete. 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 Processor -- start 2024-04-26T22:44:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 -- start start 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 0x7fbca8199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca8103570 0x7fbca8199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8103f10 0x7fbca81a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbca810ddf0 con 0x7fbca8103f10 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbca810dc70 con 0x7fbca8102370 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcae36f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbca810df70 con 0x7fbca8103570 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 0x7fbca8199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:38.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 0x7fbca8199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58548/0 (socket says 172.21.15.38:58548) 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbca7fff700 1 -- 172.21.15.38:0/1869615665 learned_addr learned my addr 172.21.15.38:0/1869615665 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbca77fe700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca8103570 0x7fbca8199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.636+0000 7fbcac90c700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8103f10 0x7fbca81a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca7fff700 1 -- 172.21.15.38:0/1869615665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca8103570 msgr2=0x7fbca8199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca7fff700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca8103570 0x7fbca8199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca7fff700 1 -- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8103f10 msgr2=0x7fbca81a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca7fff700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8103f10 0x7fbca81a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca7fff700 1 -- 172.21.15.38:0/1869615665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca81a1160 con 0x7fbca8102370 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbcac90c700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8103f10 0x7fbca81a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca7fff700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 0x7fbca8199500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800d980 tx=0x7fbc9800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc98017030 con 0x7fbca8102370 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca81a1450 con 0x7fbca8102370 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc980044d0 con 0x7fbca8102370 2024-04-26T22:44:38.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc980039e0 con 0x7fbca8102370 2024-04-26T22:44:38.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.637+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca81a1990 con 0x7fbca8102370 2024-04-26T22:44:38.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.638+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca806a110 con 0x7fbca8102370 2024-04-26T22:44:38.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.639+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fbc9801a070 con 0x7fbca8102370 2024-04-26T22:44:38.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.640+0000 7fbca57fa700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc900a3c40 0x7fbc900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:44:38.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.640+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fbc980cf3f0 con 0x7fbca8102370 2024-04-26T22:44:38.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.640+0000 7fbca77fe700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc900a3c40 0x7fbc900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:44:38.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.641+0000 7fbca77fe700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc900a3c40 0x7fbc900a6100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005950 tx=0x7fbc9c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:44:38.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.644+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc9800e350 con 0x7fbca8102370 2024-04-26T22:44:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:38 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi001 2024-04-26T22:44:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:38 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi001 to 5697M 2024-04-26T22:44:38.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:38 smithi186 ceph-mon[32774]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T22:44:38.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:38 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi001 2024-04-26T22:44:38.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:38 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi001 to 5697M 2024-04-26T22:44:38.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:38 smithi001 ceph-mon[29435]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T22:44:38.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:38 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi001 2024-04-26T22:44:38.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:38 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi001 to 5697M 2024-04-26T22:44:38.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:38 smithi038 ceph-mon[32433]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T22:44:38.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:38.862+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbca80ffaa0 con 0x7fbc900a3c40 2024-04-26T22:44:39.730 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:39 smithi038 ceph-mon[32433]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:39.730 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:39 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:44:39.730 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:39 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:44:39.730 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:39 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:39 smithi186 ceph-mon[32774]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:39 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:44:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:39 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:44:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:39 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:39 smithi001 ceph-mon[29435]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:39 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:44:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:39 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:44:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:39 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:40.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:40 smithi186 ceph-mon[32774]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:44:40.769 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:40 smithi038 ceph-mon[32433]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:44:40.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:40 smithi001 ceph-mon[29435]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:44:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:41 smithi186 ceph-mon[32774]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:41 smithi001 ceph-mon[29435]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:41 smithi038 ceph-mon[32433]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:42.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/964818913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]: dispatch 2024-04-26T22:44:42.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:42 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]: dispatch 2024-04-26T22:44:42.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:42 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]': finished 2024-04-26T22:44:42.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:42 smithi186 ceph-mon[32774]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T22:44:42.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:44:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/964818913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]: dispatch 2024-04-26T22:44:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:42 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]: dispatch 2024-04-26T22:44:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:42 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]': finished 2024-04-26T22:44:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:42 smithi001 ceph-mon[29435]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T22:44:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:44:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/964818913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]: dispatch 2024-04-26T22:44:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:42 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]: dispatch 2024-04-26T22:44:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:42 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "579e3ab3-4803-4c10-835f-266ce4f811ec"}]': finished 2024-04-26T22:44:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:42 smithi038 ceph-mon[32433]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T22:44:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:44:43.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:43 smithi186 ceph-mon[32774]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:43.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:43 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/1865913969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:44:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:43 smithi001 ceph-mon[29435]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:43 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/1865913969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:44:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:43 smithi038 ceph-mon[32433]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:43 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/1865913969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:44:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:45 smithi186 ceph-mon[32774]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:45 smithi001 ceph-mon[29435]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:45 smithi038 ceph-mon[32433]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:47 smithi186 ceph-mon[32774]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:47.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:47 smithi038 ceph-mon[32433]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:47 smithi001 ceph-mon[29435]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:49.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:49 smithi038 ceph-mon[32433]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:49.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T22:44:49.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:49 smithi186 ceph-mon[32774]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T22:44:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:49 smithi001 ceph-mon[29435]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T22:44:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:50.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:50 smithi038 ceph-mon[32433]: Deploying daemon osd.7 on smithi038 2024-04-26T22:44:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:50 smithi186 ceph-mon[32774]: Deploying daemon osd.7 on smithi038 2024-04-26T22:44:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:50 smithi001 ceph-mon[29435]: Deploying daemon osd.7 on smithi038 2024-04-26T22:44:51.564 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:51 smithi038 ceph-mon[32433]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:51.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:51 smithi186 ceph-mon[32774]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:51.746 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:51 smithi001 ceph-mon[29435]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:53 smithi186 ceph-mon[32774]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:53.767 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:53 smithi038 ceph-mon[32433]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:53 smithi001 ceph-mon[29435]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:55 smithi186 ceph-mon[32774]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:55 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:55 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:55 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:55.777 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:55 smithi038 ceph-mon[32433]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:55.777 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:55 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:55.777 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:55 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:55.777 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:55 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:55 smithi001 ceph-mon[29435]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:55 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:55 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:55 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:57.695 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:57 smithi038 ceph-mon[32433]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:57 smithi186 ceph-mon[32774]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:57 smithi001 ceph-mon[29435]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:58.530 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 7 on host 'smithi038' 2024-04-26T22:44:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.529+0000 7fbca57fa700 1 -- 172.21.15.38:0/1869615665 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbca80ffaa0 con 0x7fbc900a3c40 2024-04-26T22:44:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.531+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc900a3c40 msgr2=0x7fbc900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.531+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc900a3c40 0x7fbc900a6100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005950 tx=0x7fbc9c00b410 comp rx=0 tx=0).stop 2024-04-26T22:44:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.531+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 msgr2=0x7fbca8199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:44:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.531+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 0x7fbca8199500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800d980 tx=0x7fbc9800dc90 comp rx=0 tx=0).stop 2024-04-26T22:44:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 shutdown_connections 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc900a3c40 0x7fbc900a6100 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbca8103f10 0x7fbca81a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca8102370 0x7fbca8199500 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 --2- 172.21.15.38:0/1869615665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca8103570 0x7fbca8199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 >> 172.21.15.38:0/1869615665 conn(0x7fbca80fd910 msgr2=0x7fbca80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 shutdown_connections 2024-04-26T22:44:58.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:44:58.532+0000 7fbcae36f700 1 -- 172.21.15.38:0/1869615665 wait complete. 2024-04-26T22:44:59.182 DEBUG:teuthology.orchestra.run.smithi038:osd.7> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T22:44:59.184 INFO:tasks.cephadm:Deploying osd.8 on smithi186 with /dev/vg_nvme/lv_2... 2024-04-26T22:44:59.184 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T22:44:59.432 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:44:59.797 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:59.797 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.797 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.797 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: from='osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T22:44:59.797 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T22:44:59.798 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.798 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:44:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: from='osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:44:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: from='osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:44:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:44:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:00.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:00 smithi001 ceph-mon[29435]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T22:45:00.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:00 smithi001 ceph-mon[29435]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T22:45:00.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:00 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:00.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:00 smithi001 ceph-mon[29435]: from='osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:45:00.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:00 smithi001 ceph-mon[29435]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:45:00.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-mon[32433]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T22:45:00.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-mon[32433]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T22:45:00.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:00.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-mon[32433]: from='osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:45:00.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-mon[32433]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:45:00.852 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T22:45:00.528+0000 7f137cccd700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T22:45:00.852 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 22:45:00 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T22:45:00.538+0000 7f1378ac0700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:45:00.933 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:00 smithi186 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T22:45:00.934 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:00 smithi186 ceph-mon[32774]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T22:45:00.934 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:00 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:00.934 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:00 smithi186 ceph-mon[32774]: from='osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:45:00.934 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:00 smithi186 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:45:01.656 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T22:45:01.656 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T22:45:01.656 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T22:45:01.657 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T22:45:01.657 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113908 s, 921 MB/s 2024-04-26T22:45:01.657 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T22:45:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:01 smithi001 ceph-mon[29435]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:45:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:01 smithi001 ceph-mon[29435]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:45:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:01 smithi001 ceph-mon[29435]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T22:45:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:01.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:01 smithi038 ceph-mon[32433]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:45:01.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:01 smithi038 ceph-mon[32433]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:45:01.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:01 smithi038 ceph-mon[32433]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T22:45:01.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:01.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:01.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:01 smithi186 ceph-mon[32774]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:45:01.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:01 smithi186 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:45:01.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:01 smithi186 ceph-mon[32774]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T22:45:01.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:01.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.338 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-04-26T22:45:02.580 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] boot 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.679 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:02 smithi038 ceph-mon[32433]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] boot 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.702 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:02 smithi186 ceph-mon[32774]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: osd.7 [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] boot 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T22:45:02.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:02 smithi001 ceph-mon[29435]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T22:45:03.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:45:03.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:03.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:45:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:03.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T22:45:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:03.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:04.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 -- 172.21.15.186:0/1545974341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4102370 msgr2=0x7f64a4102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:04.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/1545974341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4102370 0x7f64a4102790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6494009990 tx=0x7f6494009ca0 comp rx=0 tx=0).stop 2024-04-26T22:45:04.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 -- 172.21.15.186:0/1545974341 shutdown_connections 2024-04-26T22:45:04.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/1545974341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4102370 0x7f64a4102790 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:04.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/1545974341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 0x7f64a410a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:04.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/1545974341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a4103570 0x7f64a41039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 -- 172.21.15.186:0/1545974341 >> 172.21.15.186:0/1545974341 conn(0x7f64a40fd950 msgr2=0x7f64a40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 -- 172.21.15.186:0/1545974341 shutdown_connections 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.274+0000 7f64ab5a9700 1 -- 172.21.15.186:0/1545974341 wait complete. 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 Processor -- start 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 -- start start 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a4102370 0x7f64a4199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 0x7f64a4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 0x7f64a41a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f64a410de70 con 0x7f64a4103f10 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64a410dcf0 con 0x7f64a4102370 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64ab5a9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64a410dff0 con 0x7f64a4103570 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64a8b44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 0x7f64a4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64a8b44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 0x7f64a4199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40708/0 (socket says 172.21.15.186:40708) 2024-04-26T22:45:04.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.275+0000 7f64a8b44700 1 -- 172.21.15.186:0/3656046545 learned_addr learned my addr 172.21.15.186:0/3656046545 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a9345700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a4102370 0x7f64a4199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a9b46700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 0x7f64a41a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a8b44700 1 -- 172.21.15.186:0/3656046545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a4102370 msgr2=0x7f64a4199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a8b44700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a4102370 0x7f64a4199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a8b44700 1 -- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 msgr2=0x7f64a41a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a8b44700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 0x7f64a41a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a8b44700 1 -- 172.21.15.186:0/3656046545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a41a12e0 con 0x7f64a4103570 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a9b46700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 0x7f64a41a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64a8b44700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 0x7f64a4199bc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64a000d9b0 tx=0x7f64a000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0016070 con 0x7f64a4103570 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a0004830 con 0x7f64a4103570 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a41a15d0 con 0x7f64a4103570 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0005500 con 0x7f64a4103570 2024-04-26T22:45:04.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.276+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a41a1b10 con 0x7f64a4103570 2024-04-26T22:45:04.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.277+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a406a110 con 0x7f64a4103570 2024-04-26T22:45:04.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.278+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f64a0004220 con 0x7f64a4103570 2024-04-26T22:45:04.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.279+0000 7f649a7fc700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f64900a3c40 0x7f64900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:04.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.279+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f64a00cfc80 con 0x7f64a4103570 2024-04-26T22:45:04.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.279+0000 7f64a9345700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f64900a3c40 0x7f64900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:04.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.280+0000 7f64a9345700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f64900a3c40 0x7f64900a6100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f64940038a0 tx=0x7f6494005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:45:04.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.283+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f64a001f020 con 0x7f64a4103570 2024-04-26T22:45:04.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:04.491+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f64a41a1dd0 con 0x7f64900a3c40 2024-04-26T22:45:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:04 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi038 2024-04-26T22:45:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:04 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-26T22:45:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:04 smithi001 ceph-mon[29435]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T22:45:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:04 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi038 2024-04-26T22:45:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:04 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-26T22:45:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:04 smithi038 ceph-mon[32433]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T22:45:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:04 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi038 2024-04-26T22:45:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:04 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-26T22:45:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:04 smithi186 ceph-mon[32774]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T22:45:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:05.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:05 smithi186 ceph-mon[32774]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:05.738 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:05 smithi186 ceph-mon[32774]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:05.738 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:05 smithi186 ceph-mon[32774]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T22:45:05.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:05 smithi001 ceph-mon[29435]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:05.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:05 smithi001 ceph-mon[29435]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:05.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:05 smithi001 ceph-mon[29435]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T22:45:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:05 smithi038 ceph-mon[32433]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:05.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:05 smithi038 ceph-mon[32433]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:05.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:05 smithi038 ceph-mon[32433]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T22:45:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:07 smithi001 ceph-mon[29435]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:07 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/3247350703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]: dispatch 2024-04-26T22:45:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:07 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]: dispatch 2024-04-26T22:45:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:07 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]': finished 2024-04-26T22:45:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:07 smithi001 ceph-mon[29435]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T22:45:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:07 smithi038 ceph-mon[32433]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:07 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/3247350703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]: dispatch 2024-04-26T22:45:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:07 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]: dispatch 2024-04-26T22:45:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:07 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]': finished 2024-04-26T22:45:07.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:07 smithi038 ceph-mon[32433]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T22:45:07.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:07 smithi186 ceph-mon[32774]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:07 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/3247350703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]: dispatch 2024-04-26T22:45:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:07 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]: dispatch 2024-04-26T22:45:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:07 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e754df8-e30f-40bf-b754-351c414e8767"}]': finished 2024-04-26T22:45:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:07 smithi186 ceph-mon[32774]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T22:45:07.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:08 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/3116925359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:45:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:08 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/3116925359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:45:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:08 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/3116925359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:45:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:09 smithi001 ceph-mon[29435]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:09 smithi038 ceph-mon[32433]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:09 smithi186 ceph-mon[32774]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:11.645 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:11 smithi186 ceph-mon[32774]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:11 smithi001 ceph-mon[29435]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:11 smithi038 ceph-mon[32433]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:13 smithi001 ceph-mon[29435]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:13 smithi038 ceph-mon[32433]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:13.953 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:13 smithi186 ceph-mon[32774]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:15 smithi001 ceph-mon[29435]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T22:45:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:15 smithi038 ceph-mon[32433]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T22:45:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:15.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:15 smithi186 ceph-mon[32774]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:15.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T22:45:15.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:16.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:16 smithi186 ceph-mon[32774]: Deploying daemon osd.8 on smithi186 2024-04-26T22:45:16.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:16 smithi001 ceph-mon[29435]: Deploying daemon osd.8 on smithi186 2024-04-26T22:45:16.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:16 smithi038 ceph-mon[32433]: Deploying daemon osd.8 on smithi186 2024-04-26T22:45:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:17 smithi001 ceph-mon[29435]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:17 smithi038 ceph-mon[32433]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:17.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:17 smithi186 ceph-mon[32774]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:19.818 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:19 smithi186 ceph-mon[32774]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:19 smithi001 ceph-mon[29435]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:19 smithi038 ceph-mon[32433]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:21 smithi186 ceph-mon[32774]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:21 smithi001 ceph-mon[29435]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:21 smithi038 ceph-mon[32433]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:23.690 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:23 smithi186 ceph-mon[32774]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:23 smithi001 ceph-mon[29435]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:23 smithi038 ceph-mon[32433]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:24.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.413+0000 7f649a7fc700 1 -- 172.21.15.186:0/3656046545 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f64a41a1dd0 con 0x7f64900a3c40 2024-04-26T22:45:24.414 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 8 on host 'smithi186' 2024-04-26T22:45:24.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f64900a3c40 msgr2=0x7f64900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f64900a3c40 0x7f64900a6100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f64940038a0 tx=0x7f6494005fb0 comp rx=0 tx=0).stop 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 msgr2=0x7f64a4199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 0x7f64a4199bc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64a000d9b0 tx=0x7f64a000dcc0 comp rx=0 tx=0).stop 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 shutdown_connections 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f64900a3c40 0x7f64900a6100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a4103f10 0x7f64a41a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a4102370 0x7f64a4199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 --2- 172.21.15.186:0/3656046545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64a4103570 0x7f64a4199bc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 >> 172.21.15.186:0/3656046545 conn(0x7f64a40fd950 msgr2=0x7f64a40ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 shutdown_connections 2024-04-26T22:45:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:45:24.416+0000 7f64ab5a9700 1 -- 172.21.15.186:0/3656046545 wait complete. 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T22:45:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T22:45:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T22:45:24.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T22:45:24.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T22:45:24.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:24.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:25.147 DEBUG:teuthology.orchestra.run.smithi186:osd.8> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T22:45:25.149 INFO:tasks.cephadm:Deploying osd.9 on smithi001 with /dev/vg_nvme/lv_1... 2024-04-26T22:45:25.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T22:45:25.406 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:45:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-mon[32774]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T22:45:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-mon[32774]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T22:45:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-mon[32774]: from='osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:45:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:45:25.752 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:25 smithi038 ceph-mon[32433]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:25.753 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:25 smithi038 ceph-mon[32433]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T22:45:25.753 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:25 smithi038 ceph-mon[32433]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T22:45:25.753 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:25 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:25.753 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:25 smithi038 ceph-mon[32433]: from='osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:45:25.753 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:25 smithi038 ceph-mon[32433]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:45:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:25 smithi001 ceph-mon[29435]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:25 smithi001 ceph-mon[29435]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T22:45:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:25 smithi001 ceph-mon[29435]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T22:45:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:25 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:25 smithi001 ceph-mon[29435]: from='osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:45:25.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:25 smithi001 ceph-mon[29435]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:45:26.237 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T22:45:25.928+0000 7fd8b2fdd700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T22:45:26.238 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 22:45:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T22:45:25.937+0000 7fd8aadc8700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:45:26.603 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:26 smithi186 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:45:26.603 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:26 smithi186 ceph-mon[32774]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T22:45:26.603 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:26 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.603 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:26 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.603 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:26 smithi186 ceph-mon[32774]: from='osd.8 ' entity='osd.8' 2024-04-26T22:45:26.603 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:26 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:26 smithi001 ceph-mon[29435]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:45:26.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:26 smithi001 ceph-mon[29435]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T22:45:26.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:26 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:26 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:26 smithi001 ceph-mon[29435]: from='osd.8 ' entity='osd.8' 2024-04-26T22:45:26.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:26 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:26 smithi038 ceph-mon[32433]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:45:26.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:26 smithi038 ceph-mon[32433]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T22:45:26.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:26 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:26 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:26.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:26 smithi038 ceph-mon[32433]: from='osd.8 ' entity='osd.8' 2024-04-26T22:45:26.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:26 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] boot 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:27 smithi186 ceph-mon[32774]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] boot 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:27.753 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:27 smithi001 ceph-mon[29435]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T22:45:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T22:45:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T22:45:27.754 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T22:45:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T22:45:27.754 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00856839 s, 1.2 GB/s 2024-04-26T22:45:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T22:45:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:45:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:45:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T22:45:27.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: osd.8 [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] boot 2024-04-26T22:45:27.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T22:45:27.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T22:45:27.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:27 smithi038 ceph-mon[32433]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T22:45:28.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_1 2024-04-26T22:45:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:45:28.945 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:28.945 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T22:45:28.945 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi186 2024-04-26T22:45:28.945 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:28.946 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:29.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi186 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:29.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi186 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T22:45:29.137 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:29.138 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:29.138 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:29.138 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 -- 172.21.15.1:0/4274396929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478103f20 msgr2=0x7fd47810a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 --2- 172.21.15.1:0/4274396929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478103f20 0x7fd47810a7d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd47000b0d0 tx=0x7fd47000b3e0 comp rx=0 tx=0).stop 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 -- 172.21.15.1:0/4274396929 shutdown_connections 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 --2- 172.21.15.1:0/4274396929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd478102380 0x7fd4781027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 --2- 172.21.15.1:0/4274396929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478103f20 0x7fd47810a7d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 --2- 172.21.15.1:0/4274396929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd478103580 0x7fd4781039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 -- 172.21.15.1:0/4274396929 >> 172.21.15.1:0/4274396929 conn(0x7fd4780fd920 msgr2=0x7fd4780ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 -- 172.21.15.1:0/4274396929 shutdown_connections 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.367+0000 7fd47f671700 1 -- 172.21.15.1:0/4274396929 wait complete. 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 Processor -- start 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 -- start start 2024-04-26T22:45:30.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 0x7fd478199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd478103580 0x7fd478199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd478103f20 0x7fd4781a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd47810de00 con 0x7fd478102380 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd47810dc80 con 0x7fd478103580 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47f671700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd47810df80 con 0x7fd478103f20 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47d40d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 0x7fd478199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47d40d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 0x7fd478199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52134/0 (socket says 172.21.15.1:52134) 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47d40d700 1 -- 172.21.15.1:0/842727853 learned_addr learned my addr 172.21.15.1:0/842727853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47cc0c700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd478103580 0x7fd478199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.368+0000 7fd47dc0e700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd478103f20 0x7fd4781a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47d40d700 1 -- 172.21.15.1:0/842727853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd478103f20 msgr2=0x7fd4781a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47d40d700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd478103f20 0x7fd4781a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47d40d700 1 -- 172.21.15.1:0/842727853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd478103580 msgr2=0x7fd478199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:30.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47d40d700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd478103580 0x7fd478199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47d40d700 1 -- 172.21.15.1:0/842727853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4781a1190 con 0x7fd478102380 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47d40d700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 0x7fd478199530 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd47400d9b0 tx=0x7fd47400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474016070 con 0x7fd478102380 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd47400be00 con 0x7fd478102380 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4781a1480 con 0x7fd478102380 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47400f3c0 con 0x7fd478102380 2024-04-26T22:45:30.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.369+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4781a1a10 con 0x7fd478102380 2024-04-26T22:45:30.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.370+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd47806a110 con 0x7fd478102380 2024-04-26T22:45:30.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.371+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd474019070 con 0x7fd478102380 2024-04-26T22:45:30.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.372+0000 7fd46e7fc700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd4640a3cc0 0x7fd4640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:30.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.372+0000 7fd47cc0c700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd4640a3cc0 0x7fd4640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:30.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.372+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fd4740cf6b0 con 0x7fd478102380 2024-04-26T22:45:30.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.372+0000 7fd47cc0c700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd4640a3cc0 0x7fd4640a6180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd46800abd0 tx=0x7fd46800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:45:30.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.377+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd474020030 con 0x7fd478102380 2024-04-26T22:45:30.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:30 smithi186 ceph-mon[32774]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T22:45:30.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:30.595+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd4780ffac0 con 0x7fd4640a3cc0 2024-04-26T22:45:30.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:30 smithi001 ceph-mon[29435]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T22:45:30.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:30 smithi038 ceph-mon[32433]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T22:45:31.480 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:31 smithi001 ceph-mon[29435]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:31.480 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:31 smithi001 ceph-mon[29435]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T22:45:31.480 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:31 smithi001 ceph-mon[29435]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:31.480 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:31.480 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:31.480 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:31 smithi186 ceph-mon[32774]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:31 smithi186 ceph-mon[32774]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T22:45:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:31 smithi186 ceph-mon[32774]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:31 smithi038 ceph-mon[32433]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:31 smithi038 ceph-mon[32433]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T22:45:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:31 smithi038 ceph-mon[32433]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:33 smithi186 ceph-mon[32774]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:33.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:33 smithi001 ceph-mon[29435]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:33.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:33 smithi038 ceph-mon[32433]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:34.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1874799556' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f781b994-298e-413f-9391-b9de190549f8"}]: dispatch 2024-04-26T22:45:34.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1874799556' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f781b994-298e-413f-9391-b9de190549f8"}]': finished 2024-04-26T22:45:34.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:34 smithi001 ceph-mon[29435]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T22:45:34.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:34 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1874799556' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f781b994-298e-413f-9391-b9de190549f8"}]: dispatch 2024-04-26T22:45:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:34 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1874799556' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f781b994-298e-413f-9391-b9de190549f8"}]': finished 2024-04-26T22:45:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:34 smithi186 ceph-mon[32774]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T22:45:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:34 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1874799556' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f781b994-298e-413f-9391-b9de190549f8"}]: dispatch 2024-04-26T22:45:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:34 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1874799556' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f781b994-298e-413f-9391-b9de190549f8"}]': finished 2024-04-26T22:45:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:34 smithi038 ceph-mon[32433]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T22:45:34.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:35 smithi186 ceph-mon[32774]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-26T22:45:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:35 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3159186115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:45:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:35 smithi001 ceph-mon[29435]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-26T22:45:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:35 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3159186115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:45:35.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:35 smithi038 ceph-mon[32433]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-26T22:45:35.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:35 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3159186115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:45:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:37 smithi038 ceph-mon[32433]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T22:45:37.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:37 smithi186 ceph-mon[32774]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T22:45:37.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:37 smithi001 ceph-mon[29435]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T22:45:39.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:39 smithi001 ceph-mon[29435]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:45:39.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:39 smithi186 ceph-mon[32774]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:45:39.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:39 smithi038 ceph-mon[32433]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:45:41.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:41 smithi001 ceph-mon[29435]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T22:45:41.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:41 smithi186 ceph-mon[32774]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T22:45:41.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:41 smithi038 ceph-mon[32433]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T22:45:42.382 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T22:45:42.383 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:42.383 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:42 smithi001 ceph-mon[29435]: Deploying daemon osd.9 on smithi001 2024-04-26T22:45:42.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T22:45:42.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:42.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:42 smithi186 ceph-mon[32774]: Deploying daemon osd.9 on smithi001 2024-04-26T22:45:42.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T22:45:42.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:42.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:42 smithi038 ceph-mon[32433]: Deploying daemon osd.9 on smithi001 2024-04-26T22:45:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:43 smithi186 ceph-mon[32774]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:45:43.525 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:43 smithi001 ceph-mon[29435]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:45:43.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:43 smithi038 ceph-mon[32433]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:45:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:45 smithi186 ceph-mon[32774]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:45.526 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:45 smithi001 ceph-mon[29435]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:45 smithi038 ceph-mon[32433]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:47 smithi186 ceph-mon[32774]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:47.512 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:47 smithi001 ceph-mon[29435]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:47 smithi038 ceph-mon[32433]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:48.466 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:48.466 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:48.467 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:48.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:48.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:48.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:49.762 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:49 smithi001 ceph-mon[29435]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:49 smithi038 ceph-mon[32433]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:49 smithi186 ceph-mon[32774]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:51.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.378+0000 7fd46e7fc700 1 -- 172.21.15.1:0/842727853 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd4780ffac0 con 0x7fd4640a3cc0 2024-04-26T22:45:51.379 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 9 on host 'smithi001' 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd4640a3cc0 msgr2=0x7fd4640a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd4640a3cc0 0x7fd4640a6180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd46800abd0 tx=0x7fd46800a380 comp rx=0 tx=0).stop 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 msgr2=0x7fd478199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 0x7fd478199530 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd47400d9b0 tx=0x7fd47400dcc0 comp rx=0 tx=0).stop 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 shutdown_connections 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd4640a3cc0 0x7fd4640a6180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd478103f20 0x7fd4781a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd478102380 0x7fd478199530 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 --2- 172.21.15.1:0/842727853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd478103580 0x7fd478199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.380+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 >> 172.21.15.1:0/842727853 conn(0x7fd4780fd920 msgr2=0x7fd4780ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:45:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.381+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 shutdown_connections 2024-04-26T22:45:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:45:51.381+0000 7fd47f671700 1 -- 172.21.15.1:0/842727853 wait complete. 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:51.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:51 smithi001 ceph-mon[29435]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T22:45:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:51.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:51 smithi038 ceph-mon[32433]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T22:45:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:51.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:51.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:51.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:51 smithi186 ceph-mon[32774]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T22:45:52.032 DEBUG:teuthology.orchestra.run.smithi001:osd.9> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T22:45:52.035 INFO:tasks.cephadm:Deploying osd.10 on smithi038 with /dev/vg_nvme/lv_1... 2024-04-26T22:45:52.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T22:45:52.290 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:45:53.427 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:53 smithi038 ceph-mon[32433]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:53.427 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:53 smithi038 ceph-mon[32433]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T22:45:53.427 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:53 smithi038 ceph-mon[32433]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T22:45:53.427 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:53 smithi038 ceph-mon[32433]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:45:53.427 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:53 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:53.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-mon[29435]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:53.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-mon[29435]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T22:45:53.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-mon[29435]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T22:45:53.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-mon[29435]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:45:53.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:53.597 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T22:45:53.374+0000 7fe614103700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T22:45:53.597 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 22:45:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T22:45:53.384+0000 7fe60fef6700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:45:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:53 smithi186 ceph-mon[32774]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:53 smithi186 ceph-mon[32774]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T22:45:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:53 smithi186 ceph-mon[32774]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T22:45:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:53 smithi186 ceph-mon[32774]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T22:45:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:53 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.371 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T22:45:54.371 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T22:45:54.371 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T22:45:54.371 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T22:45:54.371 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0158413 s, 662 MB/s 2024-04-26T22:45:54.372 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T22:45:54.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:54 smithi186 ceph-mon[32774]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:45:54.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:54 smithi186 ceph-mon[32774]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T22:45:54.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:54 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.843 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:54 smithi038 ceph-mon[32433]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:45:54.843 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:54 smithi038 ceph-mon[32433]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T22:45:54.843 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.843 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.843 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:54 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:54 smithi001 ceph-mon[29435]: from='osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T22:45:54.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:54 smithi001 ceph-mon[29435]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T22:45:54.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:54.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:54 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:55.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-04-26T22:45:55.418 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T22:45:55.541 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:55 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:45:55.541 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:55 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:45:55.541 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:55 smithi038 ceph-mon[32433]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:55.541 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:55 smithi038 ceph-mon[32433]: osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] boot 2024-04-26T22:45:55.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:55 smithi038 ceph-mon[32433]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T22:45:55.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:55 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:55 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:45:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:55 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:45:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:55 smithi186 ceph-mon[32774]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:55 smithi186 ceph-mon[32774]: osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] boot 2024-04-26T22:45:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:55 smithi186 ceph-mon[32774]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T22:45:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:55 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:55 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:45:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:55 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:45:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:55 smithi001 ceph-mon[29435]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T22:45:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:55 smithi001 ceph-mon[29435]: osd.9 [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] boot 2024-04-26T22:45:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:55 smithi001 ceph-mon[29435]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T22:45:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:55 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T22:45:56.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T22:45:56.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi001 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi001 to 4273M 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:56.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:56 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T22:45:56.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi001 2024-04-26T22:45:56.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi001 to 4273M 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:56.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:56 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T22:45:56.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi001 2024-04-26T22:45:56.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi001 to 4273M 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:56.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:56 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 -- 172.21.15.38:0/2972191491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103570 msgr2=0x7f07101039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 --2- 172.21.15.38:0/2972191491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103570 0x7f07101039d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f07000099c0 tx=0x7f0700009cd0 comp rx=0 tx=0).stop 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 -- 172.21.15.38:0/2972191491 shutdown_connections 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 --2- 172.21.15.38:0/2972191491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0710103f10 0x7f071010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 --2- 172.21.15.38:0/2972191491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0710102370 0x7f0710102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 --2- 172.21.15.38:0/2972191491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103570 0x7f07101039d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 -- 172.21.15.38:0/2972191491 >> 172.21.15.38:0/2972191491 conn(0x7f07100fd950 msgr2=0x7f07100ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 -- 172.21.15.38:0/2972191491 shutdown_connections 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 -- 172.21.15.38:0/2972191491 wait complete. 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 Processor -- start 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.321+0000 7f07182fd700 1 -- start start 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f07182fd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0710102370 0x7f0710106300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:57.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f07182fd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0710103570 0x7f0710104950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f07182fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 0x7f0710104e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f07182fd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f071010de10 con 0x7f0710102370 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f07182fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f071010dc90 con 0x7f0710103f10 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f07182fd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f071010df90 con 0x7f0710103570 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 0x7f0710104e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 0x7f0710104e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50342/0 (socket says 172.21.15.38:50342) 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 -- 172.21.15.38:0/3969218450 learned_addr learned my addr 172.21.15.38:0/3969218450 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f0715898700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0710103570 0x7f0710104950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f0716099700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0710102370 0x7f0710106300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 -- 172.21.15.38:0/3969218450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0710103570 msgr2=0x7f0710104950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0710103570 0x7f0710104950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 -- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0710102370 msgr2=0x7f0710106300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0710102370 0x7f0710106300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 -- 172.21.15.38:0/3969218450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0710105740 con 0x7f0710103f10 2024-04-26T22:45:57.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.322+0000 7f071689a700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 0x7f0710104e90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f070800ada0 tx=0x7f070800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:45:57.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.323+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708017020 con 0x7f0710103f10 2024-04-26T22:45:57.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.323+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f070800ce30 con 0x7f0710103f10 2024-04-26T22:45:57.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.323+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07101a37e0 con 0x7f0710103f10 2024-04-26T22:45:57.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.323+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708005350 con 0x7f0710103f10 2024-04-26T22:45:57.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.323+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07101a3cf0 con 0x7f0710103f10 2024-04-26T22:45:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.324+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071006a110 con 0x7f0710103f10 2024-04-26T22:45:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.325+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f070800c980 con 0x7f0710103f10 2024-04-26T22:45:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.326+0000 7f07077fe700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f06fc0a3c40 0x7f06fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:45:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.326+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f07080cf960 con 0x7f0710103f10 2024-04-26T22:45:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.326+0000 7f0716099700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f06fc0a3c40 0x7f06fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:45:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.327+0000 7f0716099700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f06fc0a3c40 0x7f06fc0a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f070c005950 tx=0x7f070c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:45:57.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.330+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0708021020 con 0x7f0710103f10 2024-04-26T22:45:57.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:45:57.554+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f071006c310 con 0x7f06fc0a3c40 2024-04-26T22:45:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:57 smithi186 ceph-mon[32774]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:45:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:57 smithi001 ceph-mon[29435]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:45:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:57 smithi038 ceph-mon[32433]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:45:58.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:58 smithi186 ceph-mon[32774]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:58.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:58.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:58.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:58 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:58.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:58 smithi001 ceph-mon[29435]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:58.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:58.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:58.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:58 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:58.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:58 smithi038 ceph-mon[32433]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:45:58.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:45:58.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:45:58.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:58 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:45:59 smithi186 ceph-mon[32774]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:45:59.743 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:45:59 smithi038 ceph-mon[32433]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:45:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:45:59 smithi001 ceph-mon[29435]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/3566819043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]: dispatch 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]: dispatch 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]': finished 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:01 smithi186 ceph-mon[32774]: from='client.? 172.21.15.38:0/1042572727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/3566819043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]: dispatch 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]: dispatch 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]': finished 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:01 smithi001 ceph-mon[29435]: from='client.? 172.21.15.38:0/1042572727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:46:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/3566819043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]: dispatch 2024-04-26T22:46:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]: dispatch 2024-04-26T22:46:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cbc770f-5853-41c8-9be5-c5b2dcec7fcc"}]': finished 2024-04-26T22:46:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T22:46:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:01 smithi038 ceph-mon[32433]: from='client.? 172.21.15.38:0/1042572727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:46:03.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:03 smithi186 ceph-mon[32774]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:03 smithi001 ceph-mon[29435]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:03 smithi038 ceph-mon[32433]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:05.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:05 smithi186 ceph-mon[32774]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:05.747 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:05 smithi038 ceph-mon[32433]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:05 smithi001 ceph-mon[29435]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:07.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:07 smithi186 ceph-mon[32774]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:07.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:07 smithi001 ceph-mon[29435]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:07.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:07 smithi038 ceph-mon[32433]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T22:46:08.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T22:46:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T22:46:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:09 smithi001 ceph-mon[29435]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:09 smithi001 ceph-mon[29435]: Deploying daemon osd.10 on smithi038 2024-04-26T22:46:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:09 smithi038 ceph-mon[32433]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:09 smithi038 ceph-mon[32433]: Deploying daemon osd.10 on smithi038 2024-04-26T22:46:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:09 smithi186 ceph-mon[32774]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:09 smithi186 ceph-mon[32774]: Deploying daemon osd.10 on smithi038 2024-04-26T22:46:11.817 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:11 smithi038 ceph-mon[32433]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:11 smithi001 ceph-mon[29435]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:11 smithi186 ceph-mon[32774]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:13.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:13 smithi038 ceph-mon[32433]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:13 smithi001 ceph-mon[29435]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:13 smithi186 ceph-mon[32774]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:15 smithi186 ceph-mon[32774]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:15 smithi001 ceph-mon[29435]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:15 smithi038 ceph-mon[32433]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:17.451 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:17 smithi038 ceph-mon[32433]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:17 smithi186 ceph-mon[32774]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:17 smithi001 ceph-mon[29435]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:18.278 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 10 on host 'smithi038' 2024-04-26T22:46:18.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.278+0000 7f07077fe700 1 -- 172.21.15.38:0/3969218450 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f071006c310 con 0x7f06fc0a3c40 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f06fc0a3c40 msgr2=0x7f06fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f06fc0a3c40 0x7f06fc0a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f070c005950 tx=0x7f070c0058e0 comp rx=0 tx=0).stop 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 msgr2=0x7f0710104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 0x7f0710104e90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f070800ada0 tx=0x7f070800c330 comp rx=0 tx=0).stop 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 shutdown_connections 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f06fc0a3c40 0x7f06fc0a6100 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0710103f10 0x7f0710104e90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0710102370 0x7f0710106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:18.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 --2- 172.21.15.38:0/3969218450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0710103570 0x7f0710104950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:18.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 >> 172.21.15.38:0/3969218450 conn(0x7f07100fd950 msgr2=0x7f071006bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:18.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 shutdown_connections 2024-04-26T22:46:18.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T22:46:18.280+0000 7f07182fd700 1 -- 172.21.15.38:0/3969218450 wait complete. 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T22:46:18.579 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T22:46:18.580 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.580 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:18 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:18.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:18.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T22:46:18.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T22:46:18.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:18 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:18.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T22:46:18.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T22:46:18.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:18 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:18.948 DEBUG:teuthology.orchestra.run.smithi038:osd.10> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T22:46:18.950 INFO:tasks.cephadm:Deploying osd.11 on smithi186 with /dev/vg_nvme/lv_1... 2024-04-26T22:46:18.950 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T22:46:19.197 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: from='osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:46:19.547 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: from='osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:46:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T22:46:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: from='osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:46:19.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T22:46:19.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:19.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T22:46:19.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:19.602 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T22:46:19.273+0000 7f51417ab700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T22:46:19.602 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 22:46:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T22:46:19.298+0000 7f513d59e700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:46:20.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:20 smithi038 ceph-mon[32433]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:46:20.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:20 smithi038 ceph-mon[32433]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T22:46:20.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:20.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:20 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:20.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:20 smithi186 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:46:20.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:20 smithi186 ceph-mon[32774]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T22:46:20.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:20.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:20 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:20.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:20 smithi001 ceph-mon[29435]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T22:46:20.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:20 smithi001 ceph-mon[29435]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T22:46:20.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:20.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:20 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:21.327 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T22:46:21.327 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T22:46:21.327 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T22:46:21.327 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T22:46:21.328 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013994 s, 749 MB/s 2024-04-26T22:46:21.328 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] boot 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T22:46:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:21.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:46:21.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:46:21.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:21.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:21.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] boot 2024-04-26T22:46:21.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T22:46:21.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: osd.10 [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] boot 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T22:46:21.631 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T22:46:22.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-04-26T22:46:22.327 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:22.589 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:22.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:22.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T22:46:22.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:23 smithi186 ceph-mon[32774]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:23 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi038 2024-04-26T22:46:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:23 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-26T22:46:23.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:23 smithi186 ceph-mon[32774]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T22:46:23.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:23 smithi001 ceph-mon[29435]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:23.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:23 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi038 2024-04-26T22:46:23.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:23 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-26T22:46:23.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:23 smithi001 ceph-mon[29435]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T22:46:23.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:23 smithi038 ceph-mon[32433]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:23.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:23 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi038 2024-04-26T22:46:23.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:23 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-26T22:46:23.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:23 smithi038 ceph-mon[32433]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T22:46:24.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 -- 172.21.15.186:0/2510782548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 msgr2=0x7f4ec8104b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:24.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/2510782548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8104b90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009990 tx=0x7f4ebc009ca0 comp rx=0 tx=0).stop 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 -- 172.21.15.186:0/2510782548 shutdown_connections 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/2510782548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 0x7f4ec810b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/2510782548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 0x7f4ec8101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/2510782548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8104b90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 -- 172.21.15.186:0/2510782548 >> 172.21.15.186:0/2510782548 conn(0x7f4ec80fd090 msgr2=0x7f4ec80ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 -- 172.21.15.186:0/2510782548 shutdown_connections 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 -- 172.21.15.186:0/2510782548 wait complete. 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 Processor -- start 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.125+0000 7f4ecc9db700 1 -- start start 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 0x7f4ec8199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 0x7f4ec81a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec810ef70 con 0x7f4ec81050d0 2024-04-26T22:46:24.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec810edf0 con 0x7f4ec8101750 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec810f0f0 con 0x7f4ec8104730 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:55208/0 (socket says 172.21.15.186:55208) 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 -- 172.21.15.186:0/3885069890 learned_addr learned my addr 172.21.15.186:0/3885069890 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec659c700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 0x7f4ec8199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec6d9d700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 0x7f4ec81a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 -- 172.21.15.186:0/3885069890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 msgr2=0x7f4ec8199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 0x7f4ec8199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 -- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 msgr2=0x7f4ec81a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 0x7f4ec81a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 -- 172.21.15.186:0/3885069890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec81a11e0 con 0x7f4ec8104730 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec659c700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 0x7f4ec8199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec6d9d700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 0x7f4ec81a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:46:24.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ec5d9b700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8199ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc003790 tx=0x7f4ebc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:24.130 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc027070 con 0x7f4ec8104730 2024-04-26T22:46:24.130 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec81a1470 con 0x7f4ec8104730 2024-04-26T22:46:24.130 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ebc022470 con 0x7f4ec8104730 2024-04-26T22:46:24.130 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00f6c0 con 0x7f4ec8104730 2024-04-26T22:46:24.130 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.126+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec81a1920 con 0x7f4ec8104730 2024-04-26T22:46:24.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.127+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec806a110 con 0x7f4ec8104730 2024-04-26T22:46:24.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.128+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4ebc02a070 con 0x7f4ec8104730 2024-04-26T22:46:24.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.129+0000 7f4eb77fe700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4eb00a3c40 0x7f4eb00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:24.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.129+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f4ebc0d0c90 con 0x7f4ec8104730 2024-04-26T22:46:24.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.129+0000 7f4ec659c700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4eb00a3c40 0x7f4eb00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:24.131 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.130+0000 7f4ec659c700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4eb00a3c40 0x7f4eb00a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8005fd0 tx=0x7f4eb8005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:24.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.134+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4ebc013070 con 0x7f4ec8104730 2024-04-26T22:46:24.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:24.343+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4ec8063860 con 0x7f4eb00a3c40 2024-04-26T22:46:24.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:24 smithi001 ceph-mon[29435]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T22:46:24.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:24 smithi038 ceph-mon[32433]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T22:46:24.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:24 smithi186 ceph-mon[32774]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T22:46:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: pgmap v242: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:46:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T22:46:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T22:46:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:46:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:46:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:46:25.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:25 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: pgmap v242: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:46:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:25 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: pgmap v242: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T22:46:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:25 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:27 smithi001 ceph-mon[29435]: pgmap v244: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:46:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:27 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/1313137131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]: dispatch 2024-04-26T22:46:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:27 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]: dispatch 2024-04-26T22:46:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:27 smithi001 ceph-mon[29435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]': finished 2024-04-26T22:46:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:27 smithi001 ceph-mon[29435]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T22:46:27.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:27 smithi038 ceph-mon[32433]: pgmap v244: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:46:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:27 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/1313137131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]: dispatch 2024-04-26T22:46:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:27 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]: dispatch 2024-04-26T22:46:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:27 smithi038 ceph-mon[32433]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]': finished 2024-04-26T22:46:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:27 smithi038 ceph-mon[32433]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T22:46:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:27 smithi186 ceph-mon[32774]: pgmap v244: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T22:46:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:27 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/1313137131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]: dispatch 2024-04-26T22:46:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:27 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]: dispatch 2024-04-26T22:46:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:27 smithi186 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e7a126e-d734-4b97-aa0e-6a547cb69575"}]': finished 2024-04-26T22:46:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:27 smithi186 ceph-mon[32774]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T22:46:27.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:28.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:28 smithi001 ceph-mon[29435]: from='client.? 172.21.15.186:0/3264737751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:46:28.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:28 smithi038 ceph-mon[32433]: from='client.? 172.21.15.186:0/3264737751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:46:28.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:28 smithi186 ceph-mon[32774]: from='client.? 172.21.15.186:0/3264737751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T22:46:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:29 smithi001 ceph-mon[29435]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T22:46:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:29 smithi001 ceph-mon[29435]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T22:46:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:29 smithi001 ceph-mon[29435]: Cluster is now healthy 2024-04-26T22:46:29.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:29 smithi038 ceph-mon[32433]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T22:46:29.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:29 smithi038 ceph-mon[32433]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T22:46:29.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:29 smithi038 ceph-mon[32433]: Cluster is now healthy 2024-04-26T22:46:29.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:29 smithi186 ceph-mon[32774]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T22:46:29.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:29 smithi186 ceph-mon[32774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T22:46:29.636 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:29 smithi186 ceph-mon[32774]: Cluster is now healthy 2024-04-26T22:46:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:31 smithi186 ceph-mon[32774]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-26T22:46:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:31 smithi001 ceph-mon[29435]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-26T22:46:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:31 smithi038 ceph-mon[32433]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-26T22:46:33.535 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:33 smithi186 ceph-mon[32774]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:46:33.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:33 smithi001 ceph-mon[29435]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:46:33.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:33 smithi038 ceph-mon[32433]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T22:46:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:35 smithi001 ceph-mon[29435]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-26T22:46:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T22:46:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:35 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:35.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:35 smithi038 ceph-mon[32433]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-26T22:46:35.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T22:46:35.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:35 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:35.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:35 smithi186 ceph-mon[32774]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-26T22:46:35.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T22:46:35.662 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:35 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:36.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:36 smithi186 ceph-mon[32774]: Deploying daemon osd.11 on smithi186 2024-04-26T22:46:36.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:36 smithi038 ceph-mon[32433]: Deploying daemon osd.11 on smithi186 2024-04-26T22:46:36.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:36 smithi001 ceph-mon[29435]: Deploying daemon osd.11 on smithi186 2024-04-26T22:46:37.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:37 smithi038 ceph-mon[32433]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:46:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:37 smithi186 ceph-mon[32774]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:46:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:37 smithi001 ceph-mon[29435]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T22:46:39.685 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:39 smithi186 ceph-mon[32774]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-26T22:46:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:39 smithi001 ceph-mon[29435]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-26T22:46:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:39 smithi038 ceph-mon[32433]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-26T22:46:41.508 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:41 smithi186 ceph-mon[32774]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:41.508 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:41.508 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:41.508 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:41 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:41 smithi001 ceph-mon[29435]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:41.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:41 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:41 smithi038 ceph-mon[32433]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:41 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:43 smithi186 ceph-mon[32774]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:43 smithi001 ceph-mon[29435]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:43 smithi038 ceph-mon[32433]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:44.363 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 11 on host 'smithi186' 2024-04-26T22:46:44.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.362+0000 7f4eb77fe700 1 -- 172.21.15.186:0/3885069890 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f4ec8063860 con 0x7f4eb00a3c40 2024-04-26T22:46:44.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.364+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4eb00a3c40 msgr2=0x7f4eb00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:44.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.364+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4eb00a3c40 0x7f4eb00a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8005fd0 tx=0x7f4eb8005de0 comp rx=0 tx=0).stop 2024-04-26T22:46:44.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.364+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 msgr2=0x7f4ec8199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:44.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.364+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8199ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc003790 tx=0x7f4ebc009f90 comp rx=0 tx=0).stop 2024-04-26T22:46:44.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 shutdown_connections 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4eb00a3c40 0x7f4eb00a6100 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ec81050d0 0x7f4ec81a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec8101750 0x7f4ec8199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 --2- 172.21.15.186:0/3885069890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ec8104730 0x7f4ec8199ac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 >> 172.21.15.186:0/3885069890 conn(0x7f4ec80fd090 msgr2=0x7f4ec8107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 shutdown_connections 2024-04-26T22:46:44.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:46:44.365+0000 7f4ecc9db700 1 -- 172.21.15.186:0/3885069890 wait complete. 2024-04-26T22:46:45.042 DEBUG:teuthology.orchestra.run.smithi186:osd.11> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T22:46:45.044 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T22:46:45.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T22:46:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: from='osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:45 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: from='osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T22:46:45.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T22:46:45.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:45 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: from='osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:45 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:46.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T22:46:46.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-mon[32774]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T22:46:46.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:46.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-mon[32774]: from='osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:46:46.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:46:46.787 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:46 smithi001 ceph-mon[29435]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T22:46:46.787 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:46 smithi001 ceph-mon[29435]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T22:46:46.787 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:46 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:46.787 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:46 smithi001 ceph-mon[29435]: from='osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:46:46.787 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:46 smithi001 ceph-mon[29435]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:46:46.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:46 smithi038 ceph-mon[32433]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T22:46:46.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:46 smithi038 ceph-mon[32433]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T22:46:46.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:46 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:46.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:46 smithi038 ceph-mon[32433]: from='osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:46:46.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:46 smithi038 ceph-mon[32433]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T22:46:47.079 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T22:46:46.793+0000 7ffbdfa72700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T22:46:47.080 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 22:46:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T22:46:46.808+0000 7ffbd785d700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T22:46:47.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 -- 172.21.15.1:0/2743659277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498103f10 msgr2=0x7f949810a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:47.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 --2- 172.21.15.1:0/2743659277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498103f10 0x7f949810a7c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f949000b0d0 tx=0x7f949000b3e0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 -- 172.21.15.1:0/2743659277 shutdown_connections 2024-04-26T22:46:47.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 --2- 172.21.15.1:0/2743659277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498102370 0x7f9498102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 --2- 172.21.15.1:0/2743659277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498103f10 0x7f949810a7c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 --2- 172.21.15.1:0/2743659277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103570 0x7f94981039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 -- 172.21.15.1:0/2743659277 >> 172.21.15.1:0/2743659277 conn(0x7f94980fd910 msgr2=0x7f94980ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 -- 172.21.15.1:0/2743659277 shutdown_connections 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.118+0000 7f94a055a700 1 -- 172.21.15.1:0/2743659277 wait complete. 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 Processor -- start 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 -- start start 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 0x7f9498199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498103570 0x7f9498199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103f10 0x7f94981a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:47.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f949810de10 con 0x7f9498102370 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f949810dc90 con 0x7f9498103f10 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f94a055a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f949810df90 con 0x7f9498103570 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f949e2f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 0x7f9498199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f949daf5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498103570 0x7f9498199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f949e2f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 0x7f9498199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40510/0 (socket says 172.21.15.1:40510) 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f949e2f6700 1 -- 172.21.15.1:0/1506974454 learned_addr learned my addr 172.21.15.1:0/1506974454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.119+0000 7f949eaf7700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103f10 0x7f94981a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949daf5700 1 -- 172.21.15.1:0/1506974454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103f10 msgr2=0x7f94981a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949daf5700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103f10 0x7f94981a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949daf5700 1 -- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 msgr2=0x7f9498199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949daf5700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 0x7f9498199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949daf5700 1 -- 172.21.15.1:0/1506974454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94981a1180 con 0x7f9498103570 2024-04-26T22:46:47.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949e2f6700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 0x7f9498199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.120+0000 7f949eaf7700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103f10 0x7f94981a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.121+0000 7f949daf5700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498103570 0x7f9498199a60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f948800ea20 tx=0x7f948800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.121+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948800cbd0 con 0x7f9498103570 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.121+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948800ee60 con 0x7f9498103570 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.121+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94981a1470 con 0x7f9498103570 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.121+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488018680 con 0x7f9498103570 2024-04-26T22:46:47.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.121+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94981a1a00 con 0x7f9498103570 2024-04-26T22:46:47.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.122+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f949806a110 con 0x7f9498103570 2024-04-26T22:46:47.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.123+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9488010430 con 0x7f9498103570 2024-04-26T22:46:47.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.124+0000 7f948f7fe700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f94840a3cc0 0x7f94840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:47.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.124+0000 7f949e2f6700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f94840a3cc0 0x7f94840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:47.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.124+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f94880cff70 con 0x7f9498103570 2024-04-26T22:46:47.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.124+0000 7f949e2f6700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f94840a3cc0 0x7f94840a6180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f94940104e0 tx=0x7f949400b450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.128+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f948809ce70 con 0x7f9498103570 2024-04-26T22:46:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.352+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f94981a2260 con 0x7f9498103570 2024-04-26T22:46:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.353+0000 7f948f7fe700 1 -- 172.21.15.1:0/1506974454 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f948809c770 con 0x7f9498103570 2024-04-26T22:46:47.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f94840a3cc0 msgr2=0x7f94840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f94840a3cc0 0x7f94840a6180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f94940104e0 tx=0x7f949400b450 comp rx=0 tx=0).stop 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498103570 msgr2=0x7f9498199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498103570 0x7f9498199a60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f948800ea20 tx=0x7f948800ed30 comp rx=0 tx=0).stop 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 shutdown_connections 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f94840a3cc0 0x7f94840a6180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9498103f10 0x7f94981a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9498102370 0x7f9498199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 --2- 172.21.15.1:0/1506974454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9498103570 0x7f9498199a60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 >> 172.21.15.1:0/1506974454 conn(0x7f94980fd910 msgr2=0x7f94980ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 shutdown_connections 2024-04-26T22:46:47.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:47.355+0000 7f94a055a700 1 -- 172.21.15.1:0/1506974454 wait complete. 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: from='osd.11 ' entity='osd.11' 2024-04-26T22:46:47.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:47 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1506974454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T22:46:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:46:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T22:46:47.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:47.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:47.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: from='osd.11 ' entity='osd.11' 2024-04-26T22:46:47.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:47 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1506974454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T22:46:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T22:46:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T22:46:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: from='osd.11 ' entity='osd.11' 2024-04-26T22:46:47.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:47 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1506974454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T22:46:47.978 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1714171580,"num_in_osds":12,"osd_in_since":1714171587,"num_remapped_pgs":0} 2024-04-26T22:46:48.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:48 smithi186 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T22:46:48.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:48 smithi186 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T22:46:48.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:48.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:48 smithi186 ceph-mon[32774]: osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] boot 2024-04-26T22:46:48.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:48 smithi186 ceph-mon[32774]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T22:46:48.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:48 smithi001 ceph-mon[29435]: purged_snaps scrub starts 2024-04-26T22:46:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:48 smithi001 ceph-mon[29435]: purged_snaps scrub ok 2024-04-26T22:46:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:48 smithi001 ceph-mon[29435]: osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] boot 2024-04-26T22:46:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:48 smithi001 ceph-mon[29435]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T22:46:48.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:48 smithi038 ceph-mon[32433]: purged_snaps scrub starts 2024-04-26T22:46:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:48 smithi038 ceph-mon[32433]: purged_snaps scrub ok 2024-04-26T22:46:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:48 smithi038 ceph-mon[32433]: osd.11 [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] boot 2024-04-26T22:46:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:48 smithi038 ceph-mon[32433]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T22:46:48.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T22:46:48.979 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T22:46:49.228 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:46:49.629 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:49.629 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.629 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:49 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:49.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:49.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:49 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T22:46:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:49.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:49 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:50.668 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:50 smithi001 ceph-mon[29435]: Detected new or changed devices on smithi186 2024-04-26T22:46:50.668 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:50 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T22:46:50.668 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:50 smithi001 ceph-mon[29435]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T22:46:50.668 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:50.668 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:50.668 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:50 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:50 smithi186 ceph-mon[32774]: Detected new or changed devices on smithi186 2024-04-26T22:46:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:50 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T22:46:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:50 smithi186 ceph-mon[32774]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T22:46:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:50.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:50 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:50 smithi038 ceph-mon[32433]: Detected new or changed devices on smithi186 2024-04-26T22:46:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:50 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T22:46:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:50 smithi038 ceph-mon[32433]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T22:46:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:50.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:50 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3702769971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 msgr2=0x7f3d141038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3702769971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d141038a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00b0d0 tx=0x7f3d0c00b3e0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3702769971 shutdown_connections 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3702769971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14105020 0x7f3d1410b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3702769971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d141038a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3702769971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14104680 0x7f3d14104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3702769971 >> 172.21.15.1:0/3702769971 conn(0x7f3d140fea20 msgr2=0x7f3d14100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3702769971 shutdown_connections 2024-04-26T22:46:51.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.221+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3702769971 wait complete. 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 Processor -- start 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 -- start start 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d1419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14104680 0x7f3d1419bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14105020 0x7f3d141a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1410ef80 con 0x7f3d14103480 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1410ee00 con 0x7f3d14105020 2024-04-26T22:46:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d1bd7a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1410f100 con 0x7f3d14104680 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.222+0000 7f3d19b16700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d1419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d1419b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40520/0 (socket says 172.21.15.1:40520) 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 -- 172.21.15.1:0/3521637602 learned_addr learned my addr 172.21.15.1:0/3521637602 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d1a317700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14105020 0x7f3d141a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19315700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14104680 0x7f3d1419bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 -- 172.21.15.1:0/3521637602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14104680 msgr2=0x7f3d1419bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14104680 0x7f3d1419bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 -- 172.21.15.1:0/3521637602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14105020 msgr2=0x7f3d141a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14105020 0x7f3d141a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 -- 172.21.15.1:0/3521637602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d141a34a0 con 0x7f3d14103480 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d1a317700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14105020 0x7f3d141a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19315700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14104680 0x7f3d1419bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d19b16700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d1419b840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c000f80 tx=0x7f3d0c004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:51.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c01f030 con 0x7f3d14103480 2024-04-26T22:46:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d141a3730 con 0x7f3d14103480 2024-04-26T22:46:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0c007b40 con 0x7f3d14103480 2024-04-26T22:46:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c004780 con 0x7f3d14103480 2024-04-26T22:46:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.223+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d141a3be0 con 0x7f3d14103480 2024-04-26T22:46:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.225+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d1406a110 con 0x7f3d14103480 2024-04-26T22:46:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.226+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f3d0c003a00 con 0x7f3d14103480 2024-04-26T22:46:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.226+0000 7f3d12ffd700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3cfc0a3c40 0x7f3cfc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.227+0000 7f3d19315700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3cfc0a3c40 0x7f3cfc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.227+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3d0c00f020 con 0x7f3d14103480 2024-04-26T22:46:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.227+0000 7f3d19315700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3cfc0a3c40 0x7f3cfc0a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d08005d90 tx=0x7f3d08005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.232+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3d0c029020 con 0x7f3d14103480 2024-04-26T22:46:51.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.440+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3d141a3f60 con 0x7f3d14103480 2024-04-26T22:46:51.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.440+0000 7f3d12ffd700 1 -- 172.21.15.1:0/3521637602 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f3d0c09e1a0 con 0x7f3d14103480 2024-04-26T22:46:51.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3cfc0a3c40 msgr2=0x7f3cfc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3cfc0a3c40 0x7f3cfc0a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d08005d90 tx=0x7f3d08005d20 comp rx=0 tx=0).stop 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 msgr2=0x7f3d1419b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d1419b840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c000f80 tx=0x7f3d0c004e00 comp rx=0 tx=0).stop 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 shutdown_connections 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3cfc0a3c40 0x7f3cfc0a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d14105020 0x7f3d141a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14103480 0x7f3d1419b840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 --2- 172.21.15.1:0/3521637602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d14104680 0x7f3d1419bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:51.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.442+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 >> 172.21.15.1:0/3521637602 conn(0x7f3d140fea20 msgr2=0x7f3d14109ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:51.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.443+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 shutdown_connections 2024-04-26T22:46:51.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:51.443+0000 7f3d1bd7a700 1 -- 172.21.15.1:0/3521637602 wait complete. 2024-04-26T22:46:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:51 smithi001 ceph-mon[29435]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:51.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:51 smithi186 ceph-mon[32774]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:51.854 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:51 smithi038 ceph-mon[32433]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:52.263 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714171607,"num_in_osds":12,"osd_in_since":1714171587,"num_remapped_pgs":0} 2024-04-26T22:46:52.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T22:46:52.511 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:46:52.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:52 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3521637602' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T22:46:52.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:52 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3521637602' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T22:46:52.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:52 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3521637602' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T22:46:53.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:53 smithi001 ceph-mon[29435]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:53 smithi186 ceph-mon[32774]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:53 smithi038 ceph-mon[32433]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:54.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/3806370536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 msgr2=0x7f2dd8106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:54.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/3806370536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd8106f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80099c0 tx=0x7f2dc8009cd0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/3806370536 shutdown_connections 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/3806370536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd81025b0 0x7f2dd8102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/3806370536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2dd8107640 0x7f2dd810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/3806370536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd8106f60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/3806370536 >> 172.21.15.1:0/3806370536 conn(0x7f2dd80792d0 msgr2=0x7f2dd80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/3806370536 shutdown_connections 2024-04-26T22:46:54.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.183+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/3806370536 wait complete. 2024-04-26T22:46:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 Processor -- start 2024-04-26T22:46:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 -- start start 2024-04-26T22:46:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2dd81025b0 0x7f2dd819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8107640 0x7f2dd8199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81124b0 con 0x7f2dd8102f60 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8112330 con 0x7f2dd81025b0 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddfc1c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8112630 con 0x7f2dd8107640 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58310/0 (socket says 172.21.15.1:58310) 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 -- 172.21.15.1:0/2106993607 learned_addr learned my addr 172.21.15.1:0/2106993607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd9b8700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2dd81025b0 0x7f2dd819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2dde1b9700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8107640 0x7f2dd8199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 -- 172.21.15.1:0/2106993607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8107640 msgr2=0x7f2dd8199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8107640 0x7f2dd8199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 -- 172.21.15.1:0/2106993607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2dd81025b0 msgr2=0x7f2dd819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2dd81025b0 0x7f2dd819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 -- 172.21.15.1:0/2106993607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd819a0e0 con 0x7f2dd8102f60 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2dde1b9700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8107640 0x7f2dd8199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:46:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.184+0000 7f2ddd1b7700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd819fd00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80045e0 tx=0x7f2dc8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.185+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8026070 con 0x7f2dd8102f60 2024-04-26T22:46:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.185+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd819a370 con 0x7f2dd8102f60 2024-04-26T22:46:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.185+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dc800f460 con 0x7f2dd8102f60 2024-04-26T22:46:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.185+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8009e60 con 0x7f2dd8102f60 2024-04-26T22:46:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.185+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd81a6690 con 0x7f2dd8102f60 2024-04-26T22:46:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.186+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd8103a40 con 0x7f2dd8102f60 2024-04-26T22:46:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.187+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f2dc8029070 con 0x7f2dd8102f60 2024-04-26T22:46:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.187+0000 7f2dceffd700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2dc40a3c40 0x7f2dc40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.188+0000 7f2ddd9b8700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2dc40a3c40 0x7f2dc40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.188+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2dc80d0910 con 0x7f2dd8102f60 2024-04-26T22:46:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.188+0000 7f2ddd9b8700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2dc40a3c40 0x7f2dc40a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2dd819aac0 tx=0x7f2dd400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:54.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.191+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2dc8030080 con 0x7f2dd8102f60 2024-04-26T22:46:54.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.402+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2dd802e490 con 0x7f2dd8102f60 2024-04-26T22:46:54.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.403+0000 7f2dceffd700 1 -- 172.21.15.1:0/2106993607 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18277 (secure 0 0 0) 0x7f2dc8013070 con 0x7f2dd8102f60 2024-04-26T22:46:54.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:46:54.404 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:46:49.379505+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2dc40a3c40 msgr2=0x7f2dc40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2dc40a3c40 0x7f2dc40a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2dd819aac0 tx=0x7f2dd400b410 comp rx=0 tx=0).stop 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 msgr2=0x7f2dd819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd819fd00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80045e0 tx=0x7f2dc8004610 comp rx=0 tx=0).stop 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 shutdown_connections 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2dc40a3c40 0x7f2dc40a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8107640 0x7f2dd8199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2dd81025b0 0x7f2dd819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 --2- 172.21.15.1:0/2106993607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd8102f60 0x7f2dd819fd00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:54.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.405+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 >> 172.21.15.1:0/2106993607 conn(0x7f2dd80792d0 msgr2=0x7f2dd806bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:54.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.406+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 shutdown_connections 2024-04-26T22:46:54.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:54.406+0000 7f2ddfc1c700 1 -- 172.21.15.1:0/2106993607 wait complete. 2024-04-26T22:46:54.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:54 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2106993607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:46:54.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:54 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2106993607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:46:54.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:54 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2106993607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:46:55.204 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:46:55.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T22:46:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:46:55.739 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:55 smithi186 ceph-mon[32774]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:55.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:55 smithi001 ceph-mon[29435]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:55.752 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:55 smithi038 ceph-mon[32433]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 -- 172.21.15.1:0/609582423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 msgr2=0x7fc63c10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 --2- 172.21.15.1:0/609582423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c10f1d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc63400b1a0 tx=0x7fc63400b4b0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 -- 172.21.15.1:0/609582423 shutdown_connections 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 --2- 172.21.15.1:0/609582423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c108390 0x7fc63c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 --2- 172.21.15.1:0/609582423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c10f1d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 --2- 172.21.15.1:0/609582423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc63c102320 0x7fc63c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 -- 172.21.15.1:0/609582423 >> 172.21.15.1:0/609582423 conn(0x7fc63c0792d0 msgr2=0x7fc63c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 -- 172.21.15.1:0/609582423 shutdown_connections 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.320+0000 7fc642661700 1 -- 172.21.15.1:0/609582423 wait complete. 2024-04-26T22:46:57.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 Processor -- start 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 -- start start 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c102320 0x7fc63c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc63c108390 0x7fc63c197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c111420 con 0x7fc63c102ce0 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c1112a0 con 0x7fc63c108390 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc642661700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c1115a0 con 0x7fc63c102320 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc633fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc633fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c19db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58332/0 (socket says 172.21.15.1:58332) 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc633fff700 1 -- 172.21.15.1:0/815482682 learned_addr learned my addr 172.21.15.1:0/815482682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.321+0000 7fc640bfe700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc63c108390 0x7fc63c197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc63bfff700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c102320 0x7fc63c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc633fff700 1 -- 172.21.15.1:0/815482682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c102320 msgr2=0x7fc63c19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc633fff700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c102320 0x7fc63c19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc633fff700 1 -- 172.21.15.1:0/815482682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc63c108390 msgr2=0x7fc63c197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc633fff700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc63c108390 0x7fc63c197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc633fff700 1 -- 172.21.15.1:0/815482682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c198000 con 0x7fc63c102ce0 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc633fff700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c19db90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc62400d950 tx=0x7fc62400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624016070 con 0x7fc63c102ce0 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc624009df0 con 0x7fc63c102ce0 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62400f3c0 con 0x7fc63c102ce0 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c1982f0 con 0x7fc63c102ce0 2024-04-26T22:46:57.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.322+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c1a45d0 con 0x7fc63c102ce0 2024-04-26T22:46:57.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.323+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c103a40 con 0x7fc63c102ce0 2024-04-26T22:46:57.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.325+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc624019070 con 0x7fc63c102ce0 2024-04-26T22:46:57.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.326+0000 7fc639ffb700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc6280a3cc0 0x7fc6280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:46:57.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.326+0000 7fc63bfff700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc6280a3cc0 0x7fc6280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:46:57.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.326+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc6240cfab0 con 0x7fc63c102ce0 2024-04-26T22:46:57.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.326+0000 7fc63bfff700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc6280a3cc0 0x7fc6280a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc62c005fd0 tx=0x7fc62c005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:46:57.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.329+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc62409c930 con 0x7fc63c102ce0 2024-04-26T22:46:57.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.538+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc63c198c50 con 0x7fc63c102ce0 2024-04-26T22:46:57.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.539+0000 7fc639ffb700 1 -- 172.21.15.1:0/815482682 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fc62409c230 con 0x7fc63c102ce0 2024-04-26T22:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:46:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc6280a3cc0 msgr2=0x7fc6280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc6280a3cc0 0x7fc6280a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc62c005fd0 tx=0x7fc62c005d90 comp rx=0 tx=0).stop 2024-04-26T22:46:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 msgr2=0x7fc63c19db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c19db90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc62400d950 tx=0x7fc62400dc60 comp rx=0 tx=0).stop 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 shutdown_connections 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc6280a3cc0 0x7fc6280a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c102320 0x7fc63c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc63c108390 0x7fc63c197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 --2- 172.21.15.1:0/815482682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c102ce0 0x7fc63c19db90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 >> 172.21.15.1:0/815482682 conn(0x7fc63c0792d0 msgr2=0x7fc63c1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 shutdown_connections 2024-04-26T22:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:46:57.541+0000 7fc642661700 1 -- 172.21.15.1:0/815482682 wait complete. 2024-04-26T22:46:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:57 smithi001 ceph-mon[29435]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:57 smithi186 ceph-mon[32774]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:57 smithi038 ceph-mon[32433]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:58.344 INFO:tasks.cephadm:Adding mds.a on smithi001 2024-04-26T22:46:58.344 INFO:tasks.cephadm:Adding mds.d on smithi001 2024-04-26T22:46:58.344 INFO:tasks.cephadm:Adding mds.g on smithi001 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.j on smithi001 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.b on smithi038 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.e on smithi038 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.h on smithi038 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.k on smithi038 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.c on smithi186 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.f on smithi186 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.i on smithi186 2024-04-26T22:46:58.345 INFO:tasks.cephadm:Adding mds.l on smithi186 2024-04-26T22:46:58.345 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l' 2024-04-26T22:46:58.596 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T22:46:58.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:58 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/815482682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:46:58.718 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:58 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/815482682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:46:58.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:58 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/815482682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:46:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:46:59 smithi186 ceph-mon[32774]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:46:59 smithi001 ceph-mon[29435]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:46:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:46:59 smithi038 ceph-mon[32433]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:00.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 -- 172.21.15.186:0/2099708417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103f10 msgr2=0x7f266810a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:00.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 --2- 172.21.15.186:0/2099708417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103f10 0x7f266810a7c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f266000b0a0 tx=0x7f266000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 -- 172.21.15.186:0/2099708417 shutdown_connections 2024-04-26T22:47:00.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 --2- 172.21.15.186:0/2099708417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103f10 0x7f266810a7c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 --2- 172.21.15.186:0/2099708417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2668102370 0x7f2668102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 --2- 172.21.15.186:0/2099708417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2668103570 0x7f26681039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 -- 172.21.15.186:0/2099708417 >> 172.21.15.186:0/2099708417 conn(0x7f26680fd910 msgr2=0x7f26680ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 -- 172.21.15.186:0/2099708417 shutdown_connections 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 -- 172.21.15.186:0/2099708417 wait complete. 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.388+0000 7f26700a4700 1 Processor -- start 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 -- start start 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2668102370 0x7f2668198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 0x7f26681988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2668103f10 0x7f266819f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f266810ddf0 con 0x7f2668103f10 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f266810dc70 con 0x7f2668102370 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f26700a4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f266810df70 con 0x7f2668103570 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 0x7f26681988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:00.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 0x7f26681988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33518/0 (socket says 172.21.15.186:33518) 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 -- 172.21.15.186:0/830764707 learned_addr learned my addr 172.21.15.186:0/830764707 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266de40700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2668102370 0x7f2668198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266e641700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2668103f10 0x7f266819f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 -- 172.21.15.186:0/830764707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2668102370 msgr2=0x7f2668198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2668102370 0x7f2668198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 -- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2668103f10 msgr2=0x7f266819f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2668103f10 0x7f266819f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 -- 172.21.15.186:0/830764707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266819ffe0 con 0x7f2668103570 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f266d63f700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 0x7f26681988c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2658007b80 tx=0x7f2658007e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.389+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265800e540 con 0x7f2668103570 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.390+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f265800eb90 con 0x7f2668103570 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.390+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265800ab50 con 0x7f2668103570 2024-04-26T22:47:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.390+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266806bbb0 con 0x7f2668103570 2024-04-26T22:47:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.390+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f266806c0f0 con 0x7f2668103570 2024-04-26T22:47:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.390+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f266806a110 con 0x7f2668103570 2024-04-26T22:47:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.392+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f265800acf0 con 0x7f2668103570 2024-04-26T22:47:00.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.392+0000 7f265effd700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f26540a3c40 0x7f26540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:00.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.392+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f26580d1840 con 0x7f2668103570 2024-04-26T22:47:00.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.392+0000 7f266de40700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f26540a3c40 0x7f26540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:00.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.393+0000 7f266de40700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f26540a3c40 0x7f26540a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f26681033d0 tx=0x7f266400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:00.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.396+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2658023020 con 0x7f2668103570 2024-04-26T22:47:00.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.604+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}) -- 0x7f266806c450 con 0x7f26540a3c40 2024-04-26T22:47:00.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.613+0000 7f265effd700 1 -- 172.21.15.186:0/830764707 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f266806c450 con 0x7f26540a3c40 2024-04-26T22:47:00.613 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mds.all update... 2024-04-26T22:47:00.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f26540a3c40 msgr2=0x7f26540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:00.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f26540a3c40 0x7f26540a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f26681033d0 tx=0x7f266400b410 comp rx=0 tx=0).stop 2024-04-26T22:47:00.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 msgr2=0x7f26681988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:00.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 0x7f26681988c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2658007b80 tx=0x7f2658007e90 comp rx=0 tx=0).stop 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 shutdown_connections 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f26540a3c40 0x7f26540a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2668103f10 0x7f266819f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2668102370 0x7f2668198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 --2- 172.21.15.186:0/830764707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2668103570 0x7f26681988c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 >> 172.21.15.186:0/830764707 conn(0x7f26680fd910 msgr2=0x7f2668106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 shutdown_connections 2024-04-26T22:47:00.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T22:47:00.615+0000 7f26700a4700 1 -- 172.21.15.186:0/830764707 wait complete. 2024-04-26T22:47:01.316 DEBUG:teuthology.orchestra.run.smithi001:mds.a> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T22:47:01.318 DEBUG:teuthology.orchestra.run.smithi001:mds.d> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T22:47:01.362 DEBUG:teuthology.orchestra.run.smithi001:mds.g> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T22:47:01.405 DEBUG:teuthology.orchestra.run.smithi001:mds.j> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T22:47:01.448 DEBUG:teuthology.orchestra.run.smithi038:mds.b> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T22:47:01.451 DEBUG:teuthology.orchestra.run.smithi038:mds.e> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T22:47:01.494 DEBUG:teuthology.orchestra.run.smithi038:mds.h> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T22:47:01.537 DEBUG:teuthology.orchestra.run.smithi038:mds.k> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T22:47:01.580 DEBUG:teuthology.orchestra.run.smithi186:mds.c> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T22:47:01.583 DEBUG:teuthology.orchestra.run.smithi186:mds.f> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T22:47:01.626 DEBUG:teuthology.orchestra.run.smithi186:mds.i> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T22:47:01.669 DEBUG:teuthology.orchestra.run.smithi186:mds.l> sudo journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T22:47:01.712 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T22:47:01.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: Saving service mds.all spec with placement smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:01.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:01 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 -- 172.21.15.1:0/2868145383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff034103470 msgr2=0x7ff034105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 --2- 172.21.15.1:0/2868145383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff034103470 0x7ff034105860 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff030009900 tx=0x7ff03002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 -- 172.21.15.1:0/2868145383 shutdown_connections 2024-04-26T22:47:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 --2- 172.21.15.1:0/2868145383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0341086d0 0x7ff03410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 --2- 172.21.15.1:0/2868145383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034105da0 0x7ff034108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:01.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 --2- 172.21.15.1:0/2868145383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff034103470 0x7ff034105860 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.828+0000 7ff03ae52640 1 -- 172.21.15.1:0/2868145383 >> 172.21.15.1:0/2868145383 conn(0x7ff0340fce40 msgr2=0x7ff0340ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 -- 172.21.15.1:0/2868145383 shutdown_connections 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 -- 172.21.15.1:0/2868145383 wait complete. 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 Processor -- start 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 -- start start 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034103470 0x7ff034198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff034105da0 0x7ff034199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 0x7ff0341a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff03410e4b0 con 0x7ff0341086d0 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff03410e330 con 0x7ff034103470 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff03ae52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff03410e630 con 0x7ff034105da0 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff0393ef640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 0x7ff0341a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff0393ef640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 0x7ff0341a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42950/0 (socket says 172.21.15.1:42950) 2024-04-26T22:47:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.829+0000 7ff0393ef640 1 -- 172.21.15.1:0/3677998110 learned_addr learned my addr 172.21.15.1:0/3677998110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff02bfff640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff034105da0 0x7ff034199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff038bee640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034103470 0x7ff034198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff0393ef640 1 -- 172.21.15.1:0/3677998110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff034105da0 msgr2=0x7ff034199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff0393ef640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff034105da0 0x7ff034199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff0393ef640 1 -- 172.21.15.1:0/3677998110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034103470 msgr2=0x7ff034198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff0393ef640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034103470 0x7ff034198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff0393ef640 1 -- 172.21.15.1:0/3677998110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0341a07c0 con 0x7ff0341086d0 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff038bee640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034103470 0x7ff034198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff0393ef640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 0x7ff0341a0150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00eaa0 tx=0x7ff02c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c00cdb0 con 0x7ff0341086d0 2024-04-26T22:47:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff02c005020 con 0x7ff0341086d0 2024-04-26T22:47:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0341a0ab0 con 0x7ff0341086d0 2024-04-26T22:47:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0116d0 con 0x7ff0341086d0 2024-04-26T22:47:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.830+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0341a0fb0 con 0x7ff0341086d0 2024-04-26T22:47:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.831+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0340fe930 con 0x7ff0341086d0 2024-04-26T22:47:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.832+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff02c0051d0 con 0x7ff0341086d0 2024-04-26T22:47:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.833+0000 7ff029ffb640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff01c0a3bb0 0x7ff01c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.833+0000 7ff038bee640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff01c0a3bb0 0x7ff01c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.833+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff02c0d02c0 con 0x7ff0341086d0 2024-04-26T22:47:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.833+0000 7ff038bee640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff01c0a3bb0 0x7ff01c0a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff030006010 tx=0x7ff030000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:01.837+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff02c016030 con 0x7ff0341086d0 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: Saving service mds.all spec with placement smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:01.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:01 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:02.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.026+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff034063530 con 0x7ff0341086d0 2024-04-26T22:47:02.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.027+0000 7ff029ffb640 1 -- 172.21.15.1:0/3677998110 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18277 (secure 0 0 0) 0x7ff02c09d0a0 con 0x7ff0341086d0 2024-04-26T22:47:02.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:02.027 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:46:49.379505+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:47:02.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff01c0a3bb0 msgr2=0x7ff01c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff01c0a3bb0 0x7ff01c0a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff030006010 tx=0x7ff030000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 msgr2=0x7ff0341a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 0x7ff0341a0150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00eaa0 tx=0x7ff02c00ef60 comp rx=0 tx=0).stop 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 shutdown_connections 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff01c0a3bb0 0x7ff01c0a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff034105da0 0x7ff034199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0341086d0 0x7ff0341a0150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 --2- 172.21.15.1:0/3677998110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff034103470 0x7ff034198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 >> 172.21.15.1:0/3677998110 conn(0x7ff0340fce40 msgr2=0x7ff034104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 shutdown_connections 2024-04-26T22:47:02.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.029+0000 7ff03ae52640 1 -- 172.21.15.1:0/3677998110 wait complete. 2024-04-26T22:47:02.039 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:02.039 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:02.039 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: Saving service mds.all spec with placement smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T22:47:02.039 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:02.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:01 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:02.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:47:02.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T22:47:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 -- 172.21.15.1:0/79858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 msgr2=0x7fae38101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 --2- 172.21.15.1:0/79858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae38101680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fae28009ca0 tx=0x7fae2802a950 comp rx=0 tx=0).stop 2024-04-26T22:47:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 -- 172.21.15.1:0/79858351 shutdown_connections 2024-04-26T22:47:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 --2- 172.21.15.1:0/79858351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae38101bc0 0x7fae38102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 --2- 172.21.15.1:0/79858351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae38102560 0x7fae381029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 --2- 172.21.15.1:0/79858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae38101680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.146+0000 7fae3d69c640 1 -- 172.21.15.1:0/79858351 >> 172.21.15.1:0/79858351 conn(0x7fae380fce00 msgr2=0x7fae380ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 -- 172.21.15.1:0/79858351 shutdown_connections 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 -- 172.21.15.1:0/79858351 wait complete. 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 Processor -- start 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 -- start start 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae3819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae38101bc0 0x7fae3819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae38102560 0x7fae381a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fae381106a0 con 0x7fae38101260 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae38110520 con 0x7fae38101bc0 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae3d69c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae38110820 con 0x7fae38102560 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.147+0000 7fae36ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae3819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae3819add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42974/0 (socket says 172.21.15.1:42974) 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 -- 172.21.15.1:0/556826290 learned_addr learned my addr 172.21.15.1:0/556826290 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae367fc640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae38101bc0 0x7fae3819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae377fe640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae38102560 0x7fae381a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 -- 172.21.15.1:0/556826290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae38102560 msgr2=0x7fae381a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae38102560 0x7fae381a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 -- 172.21.15.1:0/556826290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae38101bc0 msgr2=0x7fae3819b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae38101bc0 0x7fae3819b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 -- 172.21.15.1:0/556826290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae381a29f0 con 0x7fae38101260 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae36ffd640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae3819add0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fae28000f80 tx=0x7fae2802fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28034040 con 0x7fae38101260 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae381a2c80 con 0x7fae38101260 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae280308f0 con 0x7fae38101260 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28039450 con 0x7fae38101260 2024-04-26T22:47:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.148+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae381a31c0 con 0x7fae38101260 2024-04-26T22:47:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.149+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae3810bfb0 con 0x7fae38101260 2024-04-26T22:47:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.151+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fae28037020 con 0x7fae38101260 2024-04-26T22:47:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.152+0000 7fae2ffff640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae240a3bb0 0x7fae240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.152+0000 7fae367fc640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae240a3bb0 0x7fae240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.152+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fae280f1190 con 0x7fae38101260 2024-04-26T22:47:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.152+0000 7fae367fc640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae240a3bb0 0x7fae240a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae3819c2e0 tx=0x7fae20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.155+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fae2807f060 con 0x7fae38101260 2024-04-26T22:47:02.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.340+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fae3802dd20 con 0x7fae38101260 2024-04-26T22:47:02.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.340+0000 7fae2ffff640 1 -- 172.21.15.1:0/556826290 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fae280be010 con 0x7fae38101260 2024-04-26T22:47:02.341 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae240a3bb0 msgr2=0x7fae240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae240a3bb0 0x7fae240a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae3819c2e0 tx=0x7fae20005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 msgr2=0x7fae3819add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae3819add0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fae28000f80 tx=0x7fae2802fc20 comp rx=0 tx=0).stop 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 shutdown_connections 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae240a3bb0 0x7fae240a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae38102560 0x7fae381a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae38101bc0 0x7fae3819b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 --2- 172.21.15.1:0/556826290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae38101260 0x7fae3819add0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 >> 172.21.15.1:0/556826290 conn(0x7fae380fce00 msgr2=0x7fae38100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 shutdown_connections 2024-04-26T22:47:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.342+0000 7fae3d69c640 1 -- 172.21.15.1:0/556826290 wait complete. 2024-04-26T22:47:02.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:02.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 -- 172.21.15.1:0/669320378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc102a80 msgr2=0x7f68cc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/669320378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc102a80 0x7f68cc102f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009900 tx=0x7f68c002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 -- 172.21.15.1:0/669320378 shutdown_connections 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/669320378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68cc103440 0x7f68cc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/669320378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc102a80 0x7f68cc102f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/669320378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68cc101890 0x7f68cc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.455+0000 7f68d1bbf640 1 -- 172.21.15.1:0/669320378 >> 172.21.15.1:0/669320378 conn(0x7f68cc0fce00 msgr2=0x7f68cc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 -- 172.21.15.1:0/669320378 shutdown_connections 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 -- 172.21.15.1:0/669320378 wait complete. 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 Processor -- start 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 -- start start 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 0x7f68cc19ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68cc102a80 0x7f68cc19b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68cc103440 0x7f68cc1a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc10e4e0 con 0x7f68cc101890 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc10e360 con 0x7f68cc103440 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68d1bbf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc10e660 con 0x7f68cc102a80 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.456+0000 7f68cb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 0x7f68cc19ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 0x7f68cc19ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42996/0 (socket says 172.21.15.1:42996) 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 -- 172.21.15.1:0/2393984949 learned_addr learned my addr 172.21.15.1:0/2393984949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68caffd640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68cc102a80 0x7f68cc19b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cbfff640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68cc103440 0x7f68cc1a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 -- 172.21.15.1:0/2393984949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68cc102a80 msgr2=0x7f68cc19b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68cc102a80 0x7f68cc19b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 -- 172.21.15.1:0/2393984949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68cc103440 msgr2=0x7f68cc1a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68cc103440 0x7f68cc1a22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 -- 172.21.15.1:0/2393984949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc1a2950 con 0x7f68cc101890 2024-04-26T22:47:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68cb7fe640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 0x7f68cc19ad40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00d770 tx=0x7f68bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc017070 con 0x7f68cc101890 2024-04-26T22:47:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68bc009c30 con 0x7f68cc101890 2024-04-26T22:47:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc003cc0 con 0x7f68cc101890 2024-04-26T22:47:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc1a2c40 con 0x7f68cc101890 2024-04-26T22:47:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.457+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc1a3190 con 0x7f68cc101890 2024-04-26T22:47:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.458+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc1a3360 con 0x7f68cc101890 2024-04-26T22:47:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.459+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f68bc003e60 con 0x7f68cc101890 2024-04-26T22:47:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.460+0000 7f68c8ff9640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68b40a3bb0 0x7f68b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.460+0000 7f68caffd640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68b40a3bb0 0x7f68b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.460+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f68bc0cfe30 con 0x7f68cc101890 2024-04-26T22:47:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.460+0000 7f68caffd640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68b40a3bb0 0x7f68b40a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f68c0000c00 tx=0x7f68c00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:02.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.463+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f68bc009de0 con 0x7f68cc101890 2024-04-26T22:47:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.690+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68cc1a36f0 con 0x7f68cc101890 2024-04-26T22:47:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.690+0000 7f68c8ff9640 1 -- 172.21.15.1:0/2393984949 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f68bc09cc10 con 0x7f68cc101890 2024-04-26T22:47:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T22:47:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 1 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68b40a3bb0 msgr2=0x7f68b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68b40a3bb0 0x7f68b40a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f68c0000c00 tx=0x7f68c00314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 msgr2=0x7f68cc19ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 0x7f68cc19ad40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00d770 tx=0x7f68bc00dc30 comp rx=0 tx=0).stop 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 shutdown_connections 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f68b40a3bb0 0x7f68b40a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68cc102a80 0x7f68cc19b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68cc103440 0x7f68cc1a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 --2- 172.21.15.1:0/2393984949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68cc101890 0x7f68cc19ad40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 >> 172.21.15.1:0/2393984949 conn(0x7f68cc0fce00 msgr2=0x7f68cc10a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 shutdown_connections 2024-04-26T22:47:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:02.692+0000 7f68d1bbf640 1 -- 172.21.15.1:0/2393984949 wait complete. 2024-04-26T22:47:02.867 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:02 smithi186 ceph-mon[32774]: Deploying daemon mds.l on smithi186 2024-04-26T22:47:02.868 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:02 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3677998110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:02.868 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:02 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/556826290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:47:03.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:02 smithi001 ceph-mon[29435]: Deploying daemon mds.l on smithi186 2024-04-26T22:47:03.099 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:02 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3677998110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:03.099 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:02 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/556826290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:47:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:02 smithi038 ceph-mon[32433]: Deploying daemon mds.l on smithi186 2024-04-26T22:47:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:02 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3677998110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:02 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/556826290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:47:03.235 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 systemd[1]: Starting Ceph mds.l for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:03.521 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 podman[62237]: 2024-04-26 22:47:03.16285711 +0000 UTC m=+0.017996909 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:03.521 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 podman[62237]: 2024-04-26 22:47:03.277479536 +0000 UTC m=+0.132619368 container create 072a0005294f43fac67b2c0e48750e032062af4aaf8c4880a6b8eda4dced9686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T22:47:03.521 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 podman[62237]: 2024-04-26 22:47:03.521818584 +0000 UTC m=+0.376958373 container init 072a0005294f43fac67b2c0e48750e032062af4aaf8c4880a6b8eda4dced9686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T22:47:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-mon[32774]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2393984949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6832/4165972801,v1:172.21.15.186:6833/4165972801] up:boot 2024-04-26T22:47:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-mon[32774]: fsmap 1 up:standby 2024-04-26T22:47:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T22:47:03.985 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 podman[62237]: 2024-04-26 22:47:03.524326635 +0000 UTC m=+0.379466429 container start 072a0005294f43fac67b2c0e48750e032062af4aaf8c4880a6b8eda4dced9686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T22:47:03.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-mds[62256]: -- 172.21.15.186:0/892048854 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de075d3a40 con 0x55de075a7000 2024-04-26T22:47:03.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l[62252]: starting mds.l at 2024-04-26T22:47:03.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 bash[62237]: 072a0005294f43fac67b2c0e48750e032062af4aaf8c4880a6b8eda4dced9686 2024-04-26T22:47:03.986 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 22:47:03 smithi186 systemd[1]: Started Ceph mds.l for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:03 smithi001 ceph-mon[29435]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:03 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2393984949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:03 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6832/4165972801,v1:172.21.15.186:6833/4165972801] up:boot 2024-04-26T22:47:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:03 smithi001 ceph-mon[29435]: fsmap 1 up:standby 2024-04-26T22:47:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:03 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T22:47:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:03 smithi038 ceph-mon[32433]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:03 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2393984949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:03 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6832/4165972801,v1:172.21.15.186:6833/4165972801] up:boot 2024-04-26T22:47:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:03 smithi038 ceph-mon[32433]: fsmap 1 up:standby 2024-04-26T22:47:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:03 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T22:47:04.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 -- 172.21.15.1:0/3345165410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784103b80 msgr2=0x7f1784104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 --2- 172.21.15.1:0/3345165410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784103b80 0x7f1784104000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1774009900 tx=0x7f177402a410 comp rx=0 tx=0).stop 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 -- 172.21.15.1:0/3345165410 shutdown_connections 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 --2- 172.21.15.1:0/3345165410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1784104540 0x7f178410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 --2- 172.21.15.1:0/3345165410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784103b80 0x7f1784104000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 --2- 172.21.15.1:0/3345165410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1784102990 0x7f1784102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 -- 172.21.15.1:0/3345165410 >> 172.21.15.1:0/3345165410 conn(0x7f17840fdf20 msgr2=0x7f1784100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 -- 172.21.15.1:0/3345165410 shutdown_connections 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.819+0000 7f178c283640 1 -- 172.21.15.1:0/3345165410 wait complete. 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 Processor -- start 2024-04-26T22:47:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 -- start start 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 0x7f1784198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1784103b80 0x7f17841990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1784104540 0x7f17841a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f178410e4a0 con 0x7f1784102990 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f178410e320 con 0x7f1784103b80 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178c283640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f178410e620 con 0x7f1784104540 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178a01f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 0x7f1784198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178a01f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 0x7f1784198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43014/0 (socket says 172.21.15.1:43014) 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178a01f640 1 -- 172.21.15.1:0/88857378 learned_addr learned my addr 172.21.15.1:0/88857378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.820+0000 7f178a820640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1784104540 0x7f17841a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178981e640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1784103b80 0x7f17841990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178a01f640 1 -- 172.21.15.1:0/88857378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1784104540 msgr2=0x7f17841a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178a01f640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1784104540 0x7f17841a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178a01f640 1 -- 172.21.15.1:0/88857378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1784103b80 msgr2=0x7f17841990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178a01f640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1784103b80 0x7f17841990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178a01f640 1 -- 172.21.15.1:0/88857378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17841a0790 con 0x7f1784102990 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178a01f640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 0x7f1784198b70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f178000d950 tx=0x7f178000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780017020 con 0x7f1784102990 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17841a0a80 con 0x7f1784102990 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1780009c30 con 0x7f1784102990 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178000f6b0 con 0x7f1784102990 2024-04-26T22:47:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.821+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17841a0fd0 con 0x7f1784102990 2024-04-26T22:47:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.822+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1784109d30 con 0x7f1784102990 2024-04-26T22:47:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.823+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f178001a020 con 0x7f1784102990 2024-04-26T22:47:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.824+0000 7f177b7fe640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17700a3bb0 0x7f17700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.824+0000 7f178981e640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17700a3bb0 0x7f17700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.824+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f17800d0000 con 0x7f1784102990 2024-04-26T22:47:04.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.824+0000 7f178981e640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17700a3bb0 0x7f17700a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f178419a080 tx=0x7f17740314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:04.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:04.827+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1780009de0 con 0x7f1784102990 2024-04-26T22:47:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:04.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:04 smithi186 ceph-mon[32774]: Deploying daemon mds.k on smithi038 2024-04-26T22:47:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.054+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f178402dd40 con 0x7f1784102990 2024-04-26T22:47:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.055+0000 7f177b7fe640 1 -- 172.21.15.1:0/88857378 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f178009cde0 con 0x7f1784102990 2024-04-26T22:47:05.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:05.055 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T22:47:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 2 2024-04-26T22:47:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17700a3bb0 msgr2=0x7f17700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17700a3bb0 0x7f17700a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f178419a080 tx=0x7f17740314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 msgr2=0x7f1784198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 0x7f1784198b70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f178000d950 tx=0x7f178000de10 comp rx=0 tx=0).stop 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 shutdown_connections 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17700a3bb0 0x7f17700a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1784104540 0x7f17841a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1784103b80 0x7f17841990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 --2- 172.21.15.1:0/88857378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1784102990 0x7f1784198b70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 >> 172.21.15.1:0/88857378 conn(0x7f17840fdf20 msgr2=0x7f1784106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 shutdown_connections 2024-04-26T22:47:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:05.057+0000 7f178c283640 1 -- 172.21.15.1:0/88857378 wait complete. 2024-04-26T22:47:05.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:04 smithi001 ceph-mon[29435]: Deploying daemon mds.k on smithi038 2024-04-26T22:47:05.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:05.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:05.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:05.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:05.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:05.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:05.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:05.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:04 smithi038 ceph-mon[32433]: Deploying daemon mds.k on smithi038 2024-04-26T22:47:05.998 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:05 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/88857378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:06.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:05 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/88857378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:06.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:05 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/88857378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:06.351 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 systemd[1]: Starting Ceph mds.k for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:06.658 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 podman[61436]: 2024-04-26 22:47:06.310407612 +0000 UTC m=+0.019008476 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:06.658 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 podman[61436]: 2024-04-26 22:47:06.41990601 +0000 UTC m=+0.128506918 container create b4d7e89d01944da38811899ca801cc99754dfb851ba77a9ad27b63079af06336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T22:47:06.658 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 podman[61436]: 2024-04-26 22:47:06.655934921 +0000 UTC m=+0.364535785 container init b4d7e89d01944da38811899ca801cc99754dfb851ba77a9ad27b63079af06336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-26T22:47:06.658 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 podman[61436]: 2024-04-26 22:47:06.658631413 +0000 UTC m=+0.367232281 container start b4d7e89d01944da38811899ca801cc99754dfb851ba77a9ad27b63079af06336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T22:47:07.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:07.072 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:06 smithi038 ceph-mon[32433]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:07.072 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 ceph-mds[61456]: -- 172.21.15.38:0/604393984 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6665b7a40 con 0x55d66658ac00 2024-04-26T22:47:07.073 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k[61452]: starting mds.k at 2024-04-26T22:47:07.073 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 bash[61436]: b4d7e89d01944da38811899ca801cc99754dfb851ba77a9ad27b63079af06336 2024-04-26T22:47:07.073 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 22:47:06 smithi038 systemd[1]: Started Ceph mds.k for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:07.090 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:06 smithi001 ceph-mon[29435]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 -- 172.21.15.1:0/2801968642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 msgr2=0x7f7fdc101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/2801968642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc101cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009900 tx=0x7f7fd002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 -- 172.21.15.1:0/2801968642 shutdown_connections 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/2801968642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc103450 0x7f7fdc109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/2801968642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc102a90 0x7f7fdc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/2801968642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc101cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 -- 172.21.15.1:0/2801968642 >> 172.21.15.1:0/2801968642 conn(0x7f7fdc0fce10 msgr2=0x7f7fdc0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 -- 172.21.15.1:0/2801968642 shutdown_connections 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.172+0000 7f7fe2f90640 1 -- 172.21.15.1:0/2801968642 wait complete. 2024-04-26T22:47:07.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 Processor -- start 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 -- start start 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc102a90 0x7f7fdc19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc103450 0x7f7fdc1a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc10d3a0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc10d220 con 0x7f7fdc103450 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe2f90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc10d520 con 0x7f7fdc102a90 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc19ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43026/0 (socket says 172.21.15.1:43026) 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 -- 172.21.15.1:0/921495029 learned_addr learned my addr 172.21.15.1:0/921495029 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fdbfff640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc102a90 0x7f7fdc19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe152d640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc103450 0x7f7fdc1a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 -- 172.21.15.1:0/921495029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc102a90 msgr2=0x7f7fdc19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc102a90 0x7f7fdc19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 -- 172.21.15.1:0/921495029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc103450 msgr2=0x7f7fdc1a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc103450 0x7f7fdc1a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe0d2c640 1 -- 172.21.15.1:0/921495029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc1a2a80 con 0x7f7fdc1018a0 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fdbfff640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc102a90 0x7f7fdc19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.173+0000 7f7fe152d640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc103450 0x7f7fdc1a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.174+0000 7f7fe0d2c640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc19ae60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0006010 tx=0x7f7fd0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.174+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd003f030 con 0x7f7fdc1018a0 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.174+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd0003aa0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.174+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fdc1a2d10 con 0x7f7fdc1018a0 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.174+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0035070 con 0x7f7fdc1018a0 2024-04-26T22:47:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.174+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fdc1a31a0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.175+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fdc1a34c0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.176+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7fd0042030 con 0x7f7fdc1018a0 2024-04-26T22:47:07.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.177+0000 7f7fd9ffb640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7fcc0a3c30 0x7f7fcc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:07.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.177+0000 7f7fdbfff640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7fcc0a3c30 0x7f7fcc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:07.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.177+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7fd00f05f0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.177+0000 7f7fdbfff640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7fcc0a3c30 0x7f7fcc0a60f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc19c370 tx=0x7f7fc8005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:07.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.181+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7fd00bd3d0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:06 smithi186 ceph-mon[32774]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:07.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.413+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fdc1a3690 con 0x7f7fdc1018a0 2024-04-26T22:47:07.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.414+0000 7f7fd9ffb640 1 -- 172.21.15.1:0/921495029 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f7fd00bccd0 con 0x7f7fdc1018a0 2024-04-26T22:47:07.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:07.415 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T22:47:07.415 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7fcc0a3c30 msgr2=0x7f7fcc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7fcc0a3c30 0x7f7fcc0a60f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc19c370 tx=0x7f7fc8005ec0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 msgr2=0x7f7fdc19ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc19ae60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0006010 tx=0x7f7fd0031a50 comp rx=0 tx=0).stop 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 shutdown_connections 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7fcc0a3c30 0x7f7fcc0a60f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fdc102a90 0x7f7fdc19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fdc103450 0x7f7fdc1a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 --2- 172.21.15.1:0/921495029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1018a0 0x7f7fdc19ae60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 >> 172.21.15.1:0/921495029 conn(0x7f7fdc0fce10 msgr2=0x7f7fdc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 shutdown_connections 2024-04-26T22:47:07.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:07.416+0000 7f7fe2f90640 1 -- 172.21.15.1:0/921495029 wait complete. 2024-04-26T22:47:08.090 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6832/3943944853,v1:172.21.15.38:6833/3943944853] up:boot 2024-04-26T22:47:08.090 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: fsmap 2 up:standby 2024-04-26T22:47:08.090 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: Deploying daemon mds.j on smithi001 2024-04-26T22:47:08.091 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:07 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/921495029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6832/3943944853,v1:172.21.15.38:6833/3943944853] up:boot 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: fsmap 2 up:standby 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: Deploying daemon mds.j on smithi001 2024-04-26T22:47:08.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:07 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/921495029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:08.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6832/3943944853,v1:172.21.15.38:6833/3943944853] up:boot 2024-04-26T22:47:08.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: fsmap 2 up:standby 2024-04-26T22:47:08.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T22:47:08.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:08.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:08.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:08.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:08.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: Deploying daemon mds.j on smithi001 2024-04-26T22:47:08.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:07 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/921495029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:09.012 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:08 smithi001 ceph-mon[29435]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:09.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:08 smithi186 ceph-mon[32774]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:09.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:08 smithi038 ceph-mon[32433]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:09.346 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 systemd[1]: Starting Ceph mds.j for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:09.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 -- 172.21.15.1:0/1295998430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 msgr2=0x7fb1ec105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1295998430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec105800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009900 tx=0x7fb1e002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 -- 172.21.15.1:0/1295998430 shutdown_connections 2024-04-26T22:47:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1295998430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec108670 0x7fb1ec10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1295998430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 0x7fb1ec108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1295998430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec105800 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 -- 172.21.15.1:0/1295998430 >> 172.21.15.1:0/1295998430 conn(0x7fb1ec0fce00 msgr2=0x7fb1ec0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 -- 172.21.15.1:0/1295998430 shutdown_connections 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.536+0000 7fb1f312c640 1 -- 172.21.15.1:0/1295998430 wait complete. 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 Processor -- start 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 -- start start 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 0x7fb1ec199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec108670 0x7fb1ec1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec10e590 con 0x7fb1ec103410 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec10e410 con 0x7fb1ec105d40 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f312c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec10e710 con 0x7fb1ec108670 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43050/0 (socket says 172.21.15.1:43050) 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 -- 172.21.15.1:0/1522225191 learned_addr learned my addr 172.21.15.1:0/1522225191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1ebfff640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 0x7fb1ec199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f16c9640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec108670 0x7fb1ec1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:09.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 -- 172.21.15.1:0/1522225191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec108670 msgr2=0x7fb1ec1a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec108670 0x7fb1ec1a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 -- 172.21.15.1:0/1522225191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 msgr2=0x7fb1ec199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 0x7fb1ec199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1f0ec8640 1 -- 172.21.15.1:0/1522225191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec1a0860 con 0x7fb1ec103410 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.537+0000 7fb1ebfff640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 0x7fb1ec199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.538+0000 7fb1f0ec8640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec198c50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0005ab0 tx=0x7fb1e0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.538+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e003f070 con 0x7fb1ec103410 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.538+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e0031c50 con 0x7fb1ec103410 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.538+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec1a0af0 con 0x7fb1ec103410 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.538+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0035030 con 0x7fb1ec103410 2024-04-26T22:47:09.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.538+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec1a0fe0 con 0x7fb1ec103410 2024-04-26T22:47:09.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.539+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec0fe8f0 con 0x7fb1ec103410 2024-04-26T22:47:09.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.540+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb1e0042070 con 0x7fb1ec103410 2024-04-26T22:47:09.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.541+0000 7fb1e9ffb640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb1dc0a3bb0 0x7fb1dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:09.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.541+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb1e00f0fb0 con 0x7fb1ec103410 2024-04-26T22:47:09.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.541+0000 7fb1ebfff640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb1dc0a3bb0 0x7fb1dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:09.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.541+0000 7fb1ebfff640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb1dc0a3bb0 0x7fb1dc0a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec06a790 tx=0x7fb1d8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:09.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.544+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb1e00bde30 con 0x7fb1ec103410 2024-04-26T22:47:09.678 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 podman[68101]: 2024-04-26 22:47:09.318025269 +0000 UTC m=+0.018756967 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:09.678 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 podman[68101]: 2024-04-26 22:47:09.42290425 +0000 UTC m=+0.123635968 container create 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T22:47:09.679 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 podman[68101]: 2024-04-26 22:47:09.650812849 +0000 UTC m=+0.351544526 container init 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T22:47:09.679 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 podman[68101]: 2024-04-26 22:47:09.654082936 +0000 UTC m=+0.354814613 container start 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-26T22:47:09.679 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mds[68159]: -- 172.21.15.1:0/2656663039 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5585b02dfa40 con 0x5585b02b2800 2024-04-26T22:47:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.776+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1ec02dad0 con 0x7fb1ec103410 2024-04-26T22:47:09.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.776+0000 7fb1e9ffb640 1 -- 172.21.15.1:0/1522225191 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fb1e00bd730 con 0x7fb1ec103410 2024-04-26T22:47:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:09.777 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T22:47:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-04-26T22:47:09.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb1dc0a3bb0 msgr2=0x7fb1dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb1dc0a3bb0 0x7fb1dc0a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec06a790 tx=0x7fb1d8009110 comp rx=0 tx=0).stop 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 msgr2=0x7fb1ec198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec198c50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0005ab0 tx=0x7fb1e0003b90 comp rx=0 tx=0).stop 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 shutdown_connections 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb1dc0a3bb0 0x7fb1dc0a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec108670 0x7fb1ec1a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1ec105d40 0x7fb1ec199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 --2- 172.21.15.1:0/1522225191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1ec103410 0x7fb1ec198c50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:09.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 >> 172.21.15.1:0/1522225191 conn(0x7fb1ec0fce00 msgr2=0x7fb1ec103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 shutdown_connections 2024-04-26T22:47:09.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:09.779+0000 7fb1f312c640 1 -- 172.21.15.1:0/1522225191 wait complete. 2024-04-26T22:47:10.096 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j[68155]: starting mds.j at 2024-04-26T22:47:10.096 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 bash[68101]: 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 2024-04-26T22:47:10.096 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 22:47:09 smithi001 systemd[1]: Started Ceph mds.j for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1522225191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:10.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:10.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1522225191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1522225191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:10.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:10 smithi001 ceph-mon[29435]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:10 smithi001 ceph-mon[29435]: Deploying daemon mds.d on smithi001 2024-04-26T22:47:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:10 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6834/2653371918,v1:172.21.15.1:6835/2653371918] up:boot 2024-04-26T22:47:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:10 smithi001 ceph-mon[29435]: fsmap 3 up:standby 2024-04-26T22:47:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T22:47:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:10 smithi186 ceph-mon[32774]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:10 smithi186 ceph-mon[32774]: Deploying daemon mds.d on smithi001 2024-04-26T22:47:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:10 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6834/2653371918,v1:172.21.15.1:6835/2653371918] up:boot 2024-04-26T22:47:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:10 smithi186 ceph-mon[32774]: fsmap 3 up:standby 2024-04-26T22:47:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T22:47:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:10 smithi038 ceph-mon[32433]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:10 smithi038 ceph-mon[32433]: Deploying daemon mds.d on smithi001 2024-04-26T22:47:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:10 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6834/2653371918,v1:172.21.15.1:6835/2653371918] up:boot 2024-04-26T22:47:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:10 smithi038 ceph-mon[32433]: fsmap 3 up:standby 2024-04-26T22:47:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T22:47:11.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 -- 172.21.15.1:0/3657703996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 msgr2=0x7f9bd810aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 --2- 172.21.15.1:0/3657703996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd810aac0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009ca0 tx=0x7f9bd002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 -- 172.21.15.1:0/3657703996 shutdown_connections 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 --2- 172.21.15.1:0/3657703996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd810aac0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 --2- 172.21.15.1:0/3657703996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bd8105da0 0x7f9bd8108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 --2- 172.21.15.1:0/3657703996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd8103470 0x7f9bd8105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 -- 172.21.15.1:0/3657703996 >> 172.21.15.1:0/3657703996 conn(0x7f9bd80fce00 msgr2=0x7f9bd80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 -- 172.21.15.1:0/3657703996 shutdown_connections 2024-04-26T22:47:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.051+0000 7f9be074e640 1 -- 172.21.15.1:0/3657703996 wait complete. 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 Processor -- start 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 -- start start 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd8103470 0x7f9bd8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bd8105da0 0x7f9bd81990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd81a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd810e4e0 con 0x7f9bd81086d0 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd810e360 con 0x7f9bd8105da0 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9be074e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd810e660 con 0x7f9bd8103470 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9bdeceb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd81a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9bdeceb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd81a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49586/0 (socket says 172.21.15.1:49586) 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9bdeceb640 1 -- 172.21.15.1:0/123610657 learned_addr learned my addr 172.21.15.1:0/123610657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9bddce9640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bd8105da0 0x7f9bd81990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.052+0000 7f9bde4ea640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd8103470 0x7f9bd8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bdeceb640 1 -- 172.21.15.1:0/123610657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd8103470 msgr2=0x7f9bd8198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:12.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bdeceb640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd8103470 0x7f9bd8198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bdeceb640 1 -- 172.21.15.1:0/123610657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bd8105da0 msgr2=0x7f9bd81990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bdeceb640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bd8105da0 0x7f9bd81990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bdeceb640 1 -- 172.21.15.1:0/123610657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd81a0780 con 0x7f9bd81086d0 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bdeceb640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd81a0110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0031a20 tx=0x7f9bd0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0004240 con 0x7f9bd81086d0 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd81a0a10 con 0x7f9bd81086d0 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bd0003e10 con 0x7f9bd81086d0 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00305c0 con 0x7f9bd81086d0 2024-04-26T22:47:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.053+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd81a0f60 con 0x7f9bd81086d0 2024-04-26T22:47:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.054+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd80fe8f0 con 0x7f9bd81086d0 2024-04-26T22:47:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.056+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9bd002f660 con 0x7f9bd81086d0 2024-04-26T22:47:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.056+0000 7f9bcf7fe640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc40a3bb0 0x7f9bc40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.057+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9bd00f1770 con 0x7f9bd81086d0 2024-04-26T22:47:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.057+0000 7f9bde4ea640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc40a3bb0 0x7f9bc40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:12.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.057+0000 7f9bde4ea640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc40a3bb0 0x7f9bc40a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40058c0 tx=0x7f9bd400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.062+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9bd00be5f0 con 0x7f9bd81086d0 2024-04-26T22:47:12.096 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:11 smithi001 systemd[1]: Starting Ceph mds.d for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:12.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.299+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bd802dad0 con 0x7f9bd81086d0 2024-04-26T22:47:12.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.299+0000 7f9bcf7fe640 1 -- 172.21.15.1:0/123610657 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f9bd00bdef0 con 0x7f9bd81086d0 2024-04-26T22:47:12.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:12.300 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T22:47:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 4 2024-04-26T22:47:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc40a3bb0 msgr2=0x7f9bc40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc40a3bb0 0x7f9bc40a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40058c0 tx=0x7f9bd400a250 comp rx=0 tx=0).stop 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 msgr2=0x7f9bd81a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd81a0110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0031a20 tx=0x7f9bd0003c20 comp rx=0 tx=0).stop 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 shutdown_connections 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc40a3bb0 0x7f9bc40a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bd81086d0 0x7f9bd81a0110 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bd8105da0 0x7f9bd81990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 --2- 172.21.15.1:0/123610657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd8103470 0x7f9bd8198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 >> 172.21.15.1:0/123610657 conn(0x7f9bd80fce00 msgr2=0x7f9bd8103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 shutdown_connections 2024-04-26T22:47:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:12.301+0000 7f9be074e640 1 -- 172.21.15.1:0/123610657 wait complete. 2024-04-26T22:47:12.498 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 podman[68489]: 2024-04-26 22:47:12.113065316 +0000 UTC m=+0.019546318 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:12.499 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 podman[68489]: 2024-04-26 22:47:12.262373935 +0000 UTC m=+0.168854975 container create 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T22:47:12.499 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 podman[68489]: 2024-04-26 22:47:12.498850449 +0000 UTC m=+0.405331439 container init 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T22:47:12.846 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 podman[68489]: 2024-04-26 22:47:12.502696092 +0000 UTC m=+0.409177086 container start 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T22:47:12.846 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mds[68510]: -- 172.21.15.1:0/835251879 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b73475a40 con 0x558b73449000 2024-04-26T22:47:12.846 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d[68506]: starting mds.d at 2024-04-26T22:47:12.847 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 bash[68489]: 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 2024-04-26T22:47:12.847 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 22:47:12 smithi001 systemd[1]: Started Ceph mds.d for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/123610657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:13.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:12 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:13.342 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:13.342 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/123610657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:13.343 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.343 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.343 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.343 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:13.343 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:13.343 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:12 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/123610657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:12 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:14.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:13 smithi186 ceph-mon[32774]: Deploying daemon mds.e on smithi038 2024-04-26T22:47:14.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:13 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6836/1791309239,v1:172.21.15.1:6837/1791309239] up:boot 2024-04-26T22:47:14.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:13 smithi186 ceph-mon[32774]: fsmap 4 up:standby 2024-04-26T22:47:14.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T22:47:14.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:14.341 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:13 smithi001 ceph-mon[29435]: Deploying daemon mds.e on smithi038 2024-04-26T22:47:14.341 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:13 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6836/1791309239,v1:172.21.15.1:6837/1791309239] up:boot 2024-04-26T22:47:14.341 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:13 smithi001 ceph-mon[29435]: fsmap 4 up:standby 2024-04-26T22:47:14.341 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T22:47:14.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:13 smithi038 ceph-mon[32433]: Deploying daemon mds.e on smithi038 2024-04-26T22:47:14.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:13 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6836/1791309239,v1:172.21.15.1:6837/1791309239] up:boot 2024-04-26T22:47:14.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:13 smithi038 ceph-mon[32433]: fsmap 4 up:standby 2024-04-26T22:47:14.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 -- 172.21.15.1:0/3284639847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284102a80 msgr2=0x7f8284102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 --2- 172.21.15.1:0/3284639847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284102a80 0x7f8284102f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8270009900 tx=0x7f827002a3f0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 -- 172.21.15.1:0/3284639847 shutdown_connections 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 --2- 172.21.15.1:0/3284639847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8284103440 0x7f8284109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 --2- 172.21.15.1:0/3284639847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284102a80 0x7f8284102f00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 --2- 172.21.15.1:0/3284639847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8284101890 0x7f8284101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 -- 172.21.15.1:0/3284639847 >> 172.21.15.1:0/3284639847 conn(0x7f82840fce00 msgr2=0x7f82840ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.430+0000 7f828afaf640 1 -- 172.21.15.1:0/3284639847 shutdown_connections 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 -- 172.21.15.1:0/3284639847 wait complete. 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 Processor -- start 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 -- start start 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 0x7f8284198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8284102a80 0x7f8284199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8284103440 0x7f82841a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f828410d3a0 con 0x7f8284101890 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f828410d220 con 0x7f8284103440 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f828afaf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f828410d520 con 0x7f8284102a80 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.431+0000 7f8288d4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 0x7f8284198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 0x7f8284198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49626/0 (socket says 172.21.15.1:49626) 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 -- 172.21.15.1:0/2954656183 learned_addr learned my addr 172.21.15.1:0/2954656183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8283fff640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8284102a80 0x7f8284199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f828954c640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8284103440 0x7f82841a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 -- 172.21.15.1:0/2954656183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8284102a80 msgr2=0x7f8284199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8284102a80 0x7f8284199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 -- 172.21.15.1:0/2954656183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8284103440 msgr2=0x7f82841a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8284103440 0x7f82841a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 -- 172.21.15.1:0/2954656183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82841a0720 con 0x7f8284101890 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8288d4b640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 0x7f8284198b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f827800c970 tx=0x7f827800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278007c10 con 0x7f8284101890 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8278007db0 con 0x7f8284101890 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82841a0a10 con 0x7f8284101890 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827800f870 con 0x7f8284101890 2024-04-26T22:47:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.432+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82841a0f30 con 0x7f8284101890 2024-04-26T22:47:14.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.433+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82840697e0 con 0x7f8284101890 2024-04-26T22:47:14.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.434+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f827800fa10 con 0x7f8284101890 2024-04-26T22:47:14.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.435+0000 7f8281ffb640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82740a3bb0 0x7f82740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:14.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.435+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f82780d03b0 con 0x7f8284101890 2024-04-26T22:47:14.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.435+0000 7f8283fff640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82740a3bb0 0x7f82740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:14.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.436+0000 7f8283fff640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82740a3bb0 0x7f82740a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8270000c00 tx=0x7f82700314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:14.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.439+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f827809d220 con 0x7f8284101890 2024-04-26T22:47:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.664+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82841a1890 con 0x7f8284101890 2024-04-26T22:47:14.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.665+0000 7f8281ffb640 1 -- 172.21.15.1:0/2954656183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f827809cb20 con 0x7f8284101890 2024-04-26T22:47:14.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:14.665 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T22:47:14.666 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 5 2024-04-26T22:47:14.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82740a3bb0 msgr2=0x7f82740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:14.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82740a3bb0 0x7f82740a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8270000c00 tx=0x7f82700314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 msgr2=0x7f8284198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:14.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 0x7f8284198b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f827800c970 tx=0x7f827800ce30 comp rx=0 tx=0).stop 2024-04-26T22:47:14.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 shutdown_connections 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82740a3bb0 0x7f82740a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8284102a80 0x7f8284199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8284103440 0x7f82841a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 --2- 172.21.15.1:0/2954656183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8284101890 0x7f8284198b00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 >> 172.21.15.1:0/2954656183 conn(0x7f82840fce00 msgr2=0x7f82840fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 shutdown_connections 2024-04-26T22:47:14.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:14.667+0000 7f828afaf640 1 -- 172.21.15.1:0/2954656183 wait complete. 2024-04-26T22:47:15.083 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:14 smithi038 ceph-mon[32433]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:15.083 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:14 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2954656183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:15.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:14 smithi001 ceph-mon[29435]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:15.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:14 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2954656183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:15.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:14 smithi186 ceph-mon[32774]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:15.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:14 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2954656183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:15.351 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 podman[61758]: 2024-04-26 22:47:15.302733169 +0000 UTC m=+0.115959726 container create a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T22:47:15.851 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 podman[61758]: 2024-04-26 22:47:15.530780469 +0000 UTC m=+0.344007026 container init a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-26T22:47:15.851 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 podman[61758]: 2024-04-26 22:47:15.533474983 +0000 UTC m=+0.346701541 container start a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-26T22:47:15.851 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 ceph-mds[61777]: -- 172.21.15.38:0/708949060 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e007af3a40 con 0x55e007ac6c00 2024-04-26T22:47:15.852 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e[61773]: starting mds.e at 2024-04-26T22:47:15.852 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 bash[61758]: a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 2024-04-26T22:47:15.852 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 22:47:15 smithi038 systemd[1]: Started Ceph mds.e for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:16.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.795+0000 7f17c472b640 1 -- 172.21.15.1:0/1013368593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 msgr2=0x7f17bc105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.795+0000 7f17c472b640 1 --2- 172.21.15.1:0/1013368593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc105800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009900 tx=0x7f17b802a410 comp rx=0 tx=0).stop 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- 172.21.15.1:0/1013368593 shutdown_connections 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 --2- 172.21.15.1:0/1013368593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc105d40 0x7f17bc108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 --2- 172.21.15.1:0/1013368593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc108670 0x7f17bc10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 --2- 172.21.15.1:0/1013368593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc105800 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- 172.21.15.1:0/1013368593 >> 172.21.15.1:0/1013368593 conn(0x7f17bc0fce40 msgr2=0x7f17bc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- 172.21.15.1:0/1013368593 shutdown_connections 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- 172.21.15.1:0/1013368593 wait complete. 2024-04-26T22:47:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 Processor -- start 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- start start 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc105d40 0x7f17bc199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc108670 0x7f17bc1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc10e4f0 con 0x7f17bc103410 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc10e370 con 0x7f17bc105d40 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c472b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc10e670 con 0x7f17bc108670 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.796+0000 7f17c24c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49650/0 (socket says 172.21.15.1:49650) 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 -- 172.21.15.1:0/4041591414 learned_addr learned my addr 172.21.15.1:0/4041591414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c1cc6640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc105d40 0x7f17bc199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c2cc8640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc108670 0x7f17bc1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 -- 172.21.15.1:0/4041591414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc108670 msgr2=0x7f17bc1a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc108670 0x7f17bc1a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 -- 172.21.15.1:0/4041591414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc105d40 msgr2=0x7f17bc199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc105d40 0x7f17bc199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 -- 172.21.15.1:0/4041591414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc1a0830 con 0x7f17bc103410 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c2cc8640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc108670 0x7f17bc1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c24c7640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc198c20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f17b8005940 tx=0x7f17b8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b803f030 con 0x7f17bc103410 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b8003aa0 con 0x7f17bc103410 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17bc1a0ac0 con 0x7f17bc103410 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8035070 con 0x7f17bc103410 2024-04-26T22:47:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.797+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17bc1a0fb0 con 0x7f17bc103410 2024-04-26T22:47:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.798+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17bc0fe930 con 0x7f17bc103410 2024-04-26T22:47:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.799+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f17b8042030 con 0x7f17bc103410 2024-04-26T22:47:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.800+0000 7f17b37fe640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17a80a3bb0 0x7f17a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.800+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f17b80f0630 con 0x7f17bc103410 2024-04-26T22:47:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.800+0000 7f17c1cc6640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17a80a3bb0 0x7f17a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.800+0000 7f17c1cc6640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17a80a3bb0 0x7f17a80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f17bc06bba0 tx=0x7f17ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:16.805+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f17b80bd4b0 con 0x7f17bc103410 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: Deploying daemon mds.f on smithi186 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6834/1362256204,v1:172.21.15.38:6835/1362256204] up:boot 2024-04-26T22:47:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: fsmap 5 up:standby 2024-04-26T22:47:16.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T22:47:17.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.032+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17bc02dd20 con 0x7f17bc103410 2024-04-26T22:47:17.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.032+0000 7f17b37fe640 1 -- 172.21.15.1:0/4041591414 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f17b80bcdb0 con 0x7f17bc103410 2024-04-26T22:47:17.033 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:17.033 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T22:47:17.033 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 6 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17a80a3bb0 msgr2=0x7f17a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17a80a3bb0 0x7f17a80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f17bc06bba0 tx=0x7f17ac005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 msgr2=0x7f17bc198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc198c20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f17b8005940 tx=0x7f17b8031a50 comp rx=0 tx=0).stop 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 shutdown_connections 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f17a80a3bb0 0x7f17a80a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc108670 0x7f17bc1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc105d40 0x7f17bc199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 --2- 172.21.15.1:0/4041591414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103410 0x7f17bc198c20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.034+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 >> 172.21.15.1:0/4041591414 conn(0x7f17bc0fce40 msgr2=0x7f17bc103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.035+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 shutdown_connections 2024-04-26T22:47:17.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:17.035+0000 7f17c472b640 1 -- 172.21.15.1:0/4041591414 wait complete. 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: Deploying daemon mds.f on smithi186 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6834/1362256204,v1:172.21.15.38:6835/1362256204] up:boot 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: fsmap 5 up:standby 2024-04-26T22:47:17.045 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: Deploying daemon mds.f on smithi186 2024-04-26T22:47:17.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:17.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6834/1362256204,v1:172.21.15.38:6835/1362256204] up:boot 2024-04-26T22:47:17.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: fsmap 5 up:standby 2024-04-26T22:47:17.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T22:47:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:17 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/4041591414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:17.985 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:17 smithi186 systemd[1]: Starting Ceph mds.f for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:18.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:17 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4041591414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:18.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:17 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/4041591414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:18.292 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 podman[62541]: 2024-04-26 22:47:17.931658154 +0000 UTC m=+0.018294545 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:18.292 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 podman[62541]: 2024-04-26 22:47:18.040105964 +0000 UTC m=+0.126742324 container create c29e21bd6ef03c8d9305ff9934a628bd8221f5b2c57d32e38b2033199e8f678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T22:47:18.292 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 podman[62541]: 2024-04-26 22:47:18.292629481 +0000 UTC m=+0.379265846 container init c29e21bd6ef03c8d9305ff9934a628bd8221f5b2c57d32e38b2033199e8f678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T22:47:18.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 podman[62541]: 2024-04-26 22:47:18.295266944 +0000 UTC m=+0.381903309 container start c29e21bd6ef03c8d9305ff9934a628bd8221f5b2c57d32e38b2033199e8f678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T22:47:18.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 ceph-mds[62560]: -- 172.21.15.186:0/732578612 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561bd2359a40 con 0x561bd232c800 2024-04-26T22:47:18.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f[62556]: starting mds.f at 2024-04-26T22:47:18.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 bash[62541]: c29e21bd6ef03c8d9305ff9934a628bd8221f5b2c57d32e38b2033199e8f678e 2024-04-26T22:47:18.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 22:47:18 smithi186 systemd[1]: Started Ceph mds.f for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:19.047 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 -- 172.21.15.1:0/3169522168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44105d40 msgr2=0x7f9b44108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 --2- 172.21.15.1:0/3169522168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44105d40 0x7f9b44108130 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009900 tx=0x7f9b3802a410 comp rx=0 tx=0).stop 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 -- 172.21.15.1:0/3169522168 shutdown_connections 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 --2- 172.21.15.1:0/3169522168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b44108670 0x7f9b4410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 --2- 172.21.15.1:0/3169522168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44105d40 0x7f9b44108130 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 --2- 172.21.15.1:0/3169522168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44103410 0x7f9b44105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 -- 172.21.15.1:0/3169522168 >> 172.21.15.1:0/3169522168 conn(0x7f9b440fce00 msgr2=0x7f9b440ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 -- 172.21.15.1:0/3169522168 shutdown_connections 2024-04-26T22:47:19.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.160+0000 7f9b49103640 1 -- 172.21.15.1:0/3169522168 wait complete. 2024-04-26T22:47:19.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.161+0000 7f9b49103640 1 Processor -- start 2024-04-26T22:47:19.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.161+0000 7f9b49103640 1 -- start start 2024-04-26T22:47:19.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.161+0000 7f9b49103640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b44103410 0x7f9b44198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44105d40 0x7f9b44199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 0x7f9b441a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4410e4d0 con 0x7f9b44108670 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4410e350 con 0x7f9b44103410 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4410e650 con 0x7f9b44105d40 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 0x7f9b441a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44105d40 0x7f9b44199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44105d40 0x7f9b44199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50108/0 (socket says 172.21.15.1:50108) 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4259c640 1 -- 172.21.15.1:0/986500822 learned_addr learned my addr 172.21.15.1:0/986500822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 0x7f9b441a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49670/0 (socket says 172.21.15.1:49670) 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b42d9d640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b44103410 0x7f9b44198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 -- 172.21.15.1:0/986500822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44105d40 msgr2=0x7f9b44199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44105d40 0x7f9b44199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 -- 172.21.15.1:0/986500822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b44103410 msgr2=0x7f9b44198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b44103410 0x7f9b44198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 -- 172.21.15.1:0/986500822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b441a0840 con 0x7f9b44108670 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b4359e640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 0x7f9b441a01a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00d6b0 tx=0x7f9b3c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c004230 con 0x7f9b44108670 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c0043d0 con 0x7f9b44108670 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c003de0 con 0x7f9b44108670 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b441a0b30 con 0x7f9b44108670 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.162+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441a1030 con 0x7f9b44108670 2024-04-26T22:47:19.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.164+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b440fe8f0 con 0x7f9b44108670 2024-04-26T22:47:19.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.165+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9b3c009820 con 0x7f9b44108670 2024-04-26T22:47:19.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.165+0000 7f9b2bfff640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9b2c0a3bb0 0x7f9b2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:19.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.165+0000 7f9b42d9d640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9b2c0a3bb0 0x7f9b2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:19.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.165+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9b3c0cf9b0 con 0x7f9b44108670 2024-04-26T22:47:19.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.166+0000 7f9b42d9d640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9b2c0a3bb0 0x7f9b2c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9b4419a100 tx=0x7f9b34009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:19.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.170+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9b3c017020 con 0x7f9b44108670 2024-04-26T22:47:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.395+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b4402dd20 con 0x7f9b44108670 2024-04-26T22:47:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.396+0000 7f9b2bfff640 1 -- 172.21.15.1:0/986500822 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f9b3c09c1e0 con 0x7f9b44108670 2024-04-26T22:47:19.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:19.396 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-26T22:47:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 7 2024-04-26T22:47:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9b2c0a3bb0 msgr2=0x7f9b2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9b2c0a3bb0 0x7f9b2c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9b4419a100 tx=0x7f9b34009500 comp rx=0 tx=0).stop 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 msgr2=0x7f9b441a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 0x7f9b441a01a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00d6b0 tx=0x7f9b3c00db70 comp rx=0 tx=0).stop 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 shutdown_connections 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9b2c0a3bb0 0x7f9b2c0a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b44105d40 0x7f9b44199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b44108670 0x7f9b441a01a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 --2- 172.21.15.1:0/986500822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b44103410 0x7f9b44198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 >> 172.21.15.1:0/986500822 conn(0x7f9b440fce00 msgr2=0x7f9b44103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 shutdown_connections 2024-04-26T22:47:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:19.398+0000 7f9b49103640 1 -- 172.21.15.1:0/986500822 wait complete. 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: Deploying daemon mds.c on smithi186 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] up:boot 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: fsmap 6 up:standby 2024-04-26T22:47:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T22:47:19.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:19 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/986500822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: Deploying daemon mds.c on smithi186 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] up:boot 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: fsmap 6 up:standby 2024-04-26T22:47:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T22:47:19.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:19 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/986500822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:19.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:19.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: Deploying daemon mds.c on smithi186 2024-04-26T22:47:19.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] up:boot 2024-04-26T22:47:19.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: fsmap 6 up:standby 2024-04-26T22:47:19.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T22:47:19.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:19 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/986500822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:20.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:20 smithi186 systemd[1]: Starting Ceph mds.c for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:21.159 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:20 smithi186 podman[62847]: 2024-04-26 22:47:20.764240566 +0000 UTC m=+0.018073783 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:21.159 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:20 smithi186 podman[62847]: 2024-04-26 22:47:20.886173145 +0000 UTC m=+0.140006346 container create 2c82dc6dc1c58a02e1cf1b91822ea4630603a2e3ead2780f0f501d3de4ea8e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T22:47:21.159 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:21 smithi186 podman[62847]: 2024-04-26 22:47:21.159577169 +0000 UTC m=+0.413410370 container init 2c82dc6dc1c58a02e1cf1b91822ea4630603a2e3ead2780f0f501d3de4ea8e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T22:47:21.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:21.463 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:21 smithi186 podman[62847]: 2024-04-26 22:47:21.162455886 +0000 UTC m=+0.416289087 container start 2c82dc6dc1c58a02e1cf1b91822ea4630603a2e3ead2780f0f501d3de4ea8e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T22:47:21.463 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mds[62866]: -- 172.21.15.186:0/3666728278 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a9035da40 con 0x557a90330800 2024-04-26T22:47:21.463 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c[62862]: starting mds.c at 2024-04-26T22:47:21.463 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:21 smithi186 bash[62847]: 2c82dc6dc1c58a02e1cf1b91822ea4630603a2e3ead2780f0f501d3de4ea8e55 2024-04-26T22:47:21.464 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 22:47:21 smithi186 systemd[1]: Started Ceph mds.c for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.523+0000 7f5938c25640 1 -- 172.21.15.1:0/3340180000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934103440 msgr2=0x7f5934109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.523+0000 7f5938c25640 1 --2- 172.21.15.1:0/3340180000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934103440 0x7f5934109cd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5924009ca0 tx=0x7f592402a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 -- 172.21.15.1:0/3340180000 shutdown_connections 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 --2- 172.21.15.1:0/3340180000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5934102a80 0x7f5934102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 --2- 172.21.15.1:0/3340180000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934103440 0x7f5934109cd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 --2- 172.21.15.1:0/3340180000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934101890 0x7f5934101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 -- 172.21.15.1:0/3340180000 >> 172.21.15.1:0/3340180000 conn(0x7f59340fce00 msgr2=0x7f59340ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 -- 172.21.15.1:0/3340180000 shutdown_connections 2024-04-26T22:47:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 -- 172.21.15.1:0/3340180000 wait complete. 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 Processor -- start 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.524+0000 7f5938c25640 1 -- start start 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 0x7f593419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934102a80 0x7f593419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5934103440 0x7f59341a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f593410e4e0 con 0x7f5934101890 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f593410e360 con 0x7f5934102a80 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f593410e660 con 0x7f5934103440 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 0x7f593419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 0x7f593419ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49690/0 (socket says 172.21.15.1:49690) 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 -- 172.21.15.1:0/3578078294 learned_addr learned my addr 172.21.15.1:0/3578078294 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5931d9b640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934102a80 0x7f593419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5932d9d640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5934103440 0x7f59341a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 -- 172.21.15.1:0/3578078294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5934103440 msgr2=0x7f59341a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5934103440 0x7f59341a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 -- 172.21.15.1:0/3578078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934102a80 msgr2=0x7f593419b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934102a80 0x7f593419b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 -- 172.21.15.1:0/3578078294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59341a29b0 con 0x7f5934101890 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5931d9b640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934102a80 0x7f593419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f593259c640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 0x7f593419ada0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f592c00e9c0 tx=0x7f592c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c00cd80 con 0x7f5934101890 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59341a2ca0 con 0x7f5934101890 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f592c0035f0 con 0x7f5934101890 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c010690 con 0x7f5934101890 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.525+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59341a3190 con 0x7f5934101890 2024-04-26T22:47:21.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.526+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59341a3420 con 0x7f5934101890 2024-04-26T22:47:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.528+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f592c0037a0 con 0x7f5934101890 2024-04-26T22:47:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.528+0000 7f59237fe640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f591c0a3bb0 0x7f591c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.528+0000 7f5931d9b640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f591c0a3bb0 0x7f591c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.528+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f592c0d0330 con 0x7f5934101890 2024-04-26T22:47:21.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.529+0000 7f5931d9b640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f591c0a3bb0 0x7f591c0a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5928009fc0 tx=0x7f5928005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.533+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f592c09d1b0 con 0x7f5934101890 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:21 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.758+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59341a3760 con 0x7f5934101890 2024-04-26T22:47:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.759+0000 7f59237fe640 1 -- 172.21.15.1:0/3578078294 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7f592c09cab0 con 0x7f5934101890 2024-04-26T22:47:21.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:21.759 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T22:47:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f591c0a3bb0 msgr2=0x7f591c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f591c0a3bb0 0x7f591c0a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5928009fc0 tx=0x7f5928005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 msgr2=0x7f593419ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 0x7f593419ada0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f592c00e9c0 tx=0x7f592c00ee80 comp rx=0 tx=0).stop 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 shutdown_connections 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f591c0a3bb0 0x7f591c0a6070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5934103440 0x7f59341a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5934102a80 0x7f593419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 --2- 172.21.15.1:0/3578078294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5934101890 0x7f593419ada0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 >> 172.21.15.1:0/3578078294 conn(0x7f59340fce00 msgr2=0x7f593410a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 shutdown_connections 2024-04-26T22:47:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:21.761+0000 7f5938c25640 1 -- 172.21.15.1:0/3578078294 wait complete. 2024-04-26T22:47:21.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:21.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:21.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:21.772 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:21 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:21 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:22.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:22 smithi038 ceph-mon[32433]: Deploying daemon mds.h on smithi038 2024-04-26T22:47:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:22 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] up:boot 2024-04-26T22:47:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:22 smithi038 ceph-mon[32433]: fsmap 7 up:standby 2024-04-26T22:47:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:22 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T22:47:22.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:22 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3578078294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:22 smithi186 ceph-mon[32774]: Deploying daemon mds.h on smithi038 2024-04-26T22:47:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:22 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] up:boot 2024-04-26T22:47:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:22 smithi186 ceph-mon[32774]: fsmap 7 up:standby 2024-04-26T22:47:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:22 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T22:47:22.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:22 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3578078294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:22.845 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:22 smithi001 ceph-mon[29435]: Deploying daemon mds.h on smithi038 2024-04-26T22:47:22.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:22 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] up:boot 2024-04-26T22:47:22.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:22 smithi001 ceph-mon[29435]: fsmap 7 up:standby 2024-04-26T22:47:22.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:22 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T22:47:22.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:22 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3578078294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:23.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:23 smithi038 ceph-mon[32433]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:23 smithi186 ceph-mon[32774]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:23.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:23.802 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:23 smithi001 ceph-mon[29435]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:23.851 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:23 smithi038 systemd[1]: Starting Ceph mds.h for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 -- 172.21.15.1:0/406038944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364108670 msgr2=0x7fe36410aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 --2- 172.21.15.1:0/406038944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364108670 0x7fe36410aac0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009ca0 tx=0x7fe35c02a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 -- 172.21.15.1:0/406038944 shutdown_connections 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 --2- 172.21.15.1:0/406038944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364108670 0x7fe36410aac0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 --2- 172.21.15.1:0/406038944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe364105d40 0x7fe364108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 --2- 172.21.15.1:0/406038944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe364103410 0x7fe364105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 -- 172.21.15.1:0/406038944 >> 172.21.15.1:0/406038944 conn(0x7fe3640fce40 msgr2=0x7fe3640ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 -- 172.21.15.1:0/406038944 shutdown_connections 2024-04-26T22:47:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.891+0000 7fe368dc1640 1 -- 172.21.15.1:0/406038944 wait complete. 2024-04-26T22:47:23.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 Processor -- start 2024-04-26T22:47:23.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 -- start start 2024-04-26T22:47:23.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe364103410 0x7fe364198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:23.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 0x7fe3641990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe364108670 0x7fe3641a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe36410e530 con 0x7fe364105d40 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36410e3b0 con 0x7fe364108670 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe368dc1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe36410e6b0 con 0x7fe364103410 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 0x7fe3641990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 0x7fe3641990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48412/0 (socket says 172.21.15.1:48412) 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 -- 172.21.15.1:0/2897483160 learned_addr learned my addr 172.21.15.1:0/2897483160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe362d9d640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe364108670 0x7fe3641a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe36259c640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe364103410 0x7fe364198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 -- 172.21.15.1:0/2897483160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe364103410 msgr2=0x7fe364198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe364103410 0x7fe364198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:23.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 -- 172.21.15.1:0/2897483160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe364108670 msgr2=0x7fe3641a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe364108670 0x7fe3641a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.892+0000 7fe361d9b640 1 -- 172.21.15.1:0/2897483160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3641a0790 con 0x7fe364105d40 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.893+0000 7fe361d9b640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 0x7fe3641990b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe35800e9c0 tx=0x7fe35800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.893+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35800cd80 con 0x7fe364105d40 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.893+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3641a0a80 con 0x7fe364105d40 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.893+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3580054d0 con 0x7fe364105d40 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.893+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358010430 con 0x7fe364105d40 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.893+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3641a1020 con 0x7fe364105d40 2024-04-26T22:47:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.894+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3640fe930 con 0x7fe364105d40 2024-04-26T22:47:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.896+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe358004fb0 con 0x7fe364105d40 2024-04-26T22:47:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.896+0000 7fe3537fe640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe34c0a3bb0 0x7fe34c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.896+0000 7fe36259c640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe34c0a3bb0 0x7fe34c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.896+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe3580d05f0 con 0x7fe364105d40 2024-04-26T22:47:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.897+0000 7fe36259c640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe34c0a3bb0 0x7fe34c0a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe354005fd0 tx=0x7fe354005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:23.900+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe358054060 con 0x7fe364105d40 2024-04-26T22:47:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.124+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe36402dad0 con 0x7fe364105d40 2024-04-26T22:47:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.125+0000 7fe3537fe640 1 -- 172.21.15.1:0/2897483160 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7fe35809d470 con 0x7fe364105d40 2024-04-26T22:47:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:24.126 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T22:47:24.126 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe34c0a3bb0 msgr2=0x7fe34c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe34c0a3bb0 0x7fe34c0a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe354005fd0 tx=0x7fe354005850 comp rx=0 tx=0).stop 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 msgr2=0x7fe3641990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 0x7fe3641990b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe35800e9c0 tx=0x7fe35800ee80 comp rx=0 tx=0).stop 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 shutdown_connections 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe34c0a3bb0 0x7fe34c0a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe364108670 0x7fe3641a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe364105d40 0x7fe3641990b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 --2- 172.21.15.1:0/2897483160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe364103410 0x7fe364198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.127+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 >> 172.21.15.1:0/2897483160 conn(0x7fe3640fce40 msgr2=0x7fe364103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.128+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 shutdown_connections 2024-04-26T22:47:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:24.128+0000 7fe368dc1640 1 -- 172.21.15.1:0/2897483160 wait complete. 2024-04-26T22:47:24.254 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:23 smithi038 podman[62067]: 2024-04-26 22:47:23.892697176 +0000 UTC m=+0.018845940 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:24.254 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 podman[62067]: 2024-04-26 22:47:24.010168676 +0000 UTC m=+0.136317450 container create da7eb814513118fd181d2f9105094cb0c1c1b09d0b4057d955c0154138115c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T22:47:24.254 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 podman[62067]: 2024-04-26 22:47:24.254524901 +0000 UTC m=+0.380673634 container init da7eb814513118fd181d2f9105094cb0c1c1b09d0b4057d955c0154138115c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T22:47:24.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2897483160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:24.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 podman[62067]: 2024-04-26 22:47:24.257272604 +0000 UTC m=+0.383421337 container start da7eb814513118fd181d2f9105094cb0c1c1b09d0b4057d955c0154138115c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-mds[62086]: -- 172.21.15.38:0/1602237979 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55584a6b7a40 con 0x55584a68a800 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h[62082]: starting mds.h at 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 bash[62067]: da7eb814513118fd181d2f9105094cb0c1c1b09d0b4057d955c0154138115c20 2024-04-26T22:47:24.602 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 22:47:24 smithi038 systemd[1]: Started Ceph mds.h for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2897483160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:24.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:24 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2897483160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:24 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:25 smithi186 ceph-mon[32774]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:25 smithi186 ceph-mon[32774]: Deploying daemon mds.i on smithi186 2024-04-26T22:47:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:25 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] up:boot 2024-04-26T22:47:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:25 smithi186 ceph-mon[32774]: fsmap 8 up:standby 2024-04-26T22:47:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:25 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T22:47:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:25 smithi001 ceph-mon[29435]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:25 smithi001 ceph-mon[29435]: Deploying daemon mds.i on smithi186 2024-04-26T22:47:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:25 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] up:boot 2024-04-26T22:47:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:25 smithi001 ceph-mon[29435]: fsmap 8 up:standby 2024-04-26T22:47:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:25 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T22:47:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:25 smithi038 ceph-mon[32433]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:25 smithi038 ceph-mon[32433]: Deploying daemon mds.i on smithi186 2024-04-26T22:47:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:25 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] up:boot 2024-04-26T22:47:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:25 smithi038 ceph-mon[32433]: fsmap 8 up:standby 2024-04-26T22:47:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:25 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T22:47:26.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 -- 172.21.15.1:0/475315818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 msgr2=0x7ff6dc10add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 --2- 172.21.15.1:0/475315818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc10add0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009ca0 tx=0x7ff6d402a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 -- 172.21.15.1:0/475315818 shutdown_connections 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 --2- 172.21.15.1:0/475315818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6dc103b80 0x7ff6dc104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 --2- 172.21.15.1:0/475315818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc10add0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 --2- 172.21.15.1:0/475315818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6dc102990 0x7ff6dc102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 -- 172.21.15.1:0/475315818 >> 172.21.15.1:0/475315818 conn(0x7ff6dc0fdf20 msgr2=0x7ff6dc100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 -- 172.21.15.1:0/475315818 shutdown_connections 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.254+0000 7ff6e190a640 1 -- 172.21.15.1:0/475315818 wait complete. 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 Processor -- start 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 -- start start 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6dc102990 0x7ff6dc0766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6dc103b80 0x7ff6dc074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc10e4c0 con 0x7ff6dc104540 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc10e340 con 0x7ff6dc103b80 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6e190a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc10e640 con 0x7ff6dc102990 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6db7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc075260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48426/0 (socket says 172.21.15.1:48426) 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6db7fe640 1 -- 172.21.15.1:0/123613449 learned_addr learned my addr 172.21.15.1:0/123613449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6da7fc640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6dc103b80 0x7ff6dc074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.255+0000 7ff6daffd640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6dc102990 0x7ff6dc0766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6db7fe640 1 -- 172.21.15.1:0/123613449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6dc102990 msgr2=0x7ff6dc0766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6db7fe640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6dc102990 0x7ff6dc0766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6db7fe640 1 -- 172.21.15.1:0/123613449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6dc103b80 msgr2=0x7ff6dc074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6db7fe640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6dc103b80 0x7ff6dc074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6db7fe640 1 -- 172.21.15.1:0/123613449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc075af0 con 0x7ff6dc104540 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6db7fe640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc075260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40097b0 tx=0x7ff6d4031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4003bf0 con 0x7ff6dc104540 2024-04-26T22:47:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6d4003d90 con 0x7ff6dc104540 2024-04-26T22:47:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1a2c90 con 0x7ff6dc104540 2024-04-26T22:47:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d402fc70 con 0x7ff6dc104540 2024-04-26T22:47:26.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.256+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1a3190 con 0x7ff6dc104540 2024-04-26T22:47:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.257+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc109cb0 con 0x7ff6dc104540 2024-04-26T22:47:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.258+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff6d4004b40 con 0x7ff6dc104540 2024-04-26T22:47:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.259+0000 7ff6e0908640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff6c40a3bb0 0x7ff6c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.259+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff6d40f17d0 con 0x7ff6dc104540 2024-04-26T22:47:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.259+0000 7ff6daffd640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff6c40a3bb0 0x7ff6c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:26.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.259+0000 7ff6daffd640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff6c40a3bb0 0x7ff6c40a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc005d00 tx=0x7ff6cc005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:26.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.262+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff6d40be650 con 0x7ff6dc104540 2024-04-26T22:47:26.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.487+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6dc02dad0 con 0x7ff6dc104540 2024-04-26T22:47:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.488+0000 7ff6e0908640 1 -- 172.21.15.1:0/123613449 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7ff6d40bdf50 con 0x7ff6dc104540 2024-04-26T22:47:26.488 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:26.489 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T22:47:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff6c40a3bb0 msgr2=0x7ff6c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff6c40a3bb0 0x7ff6c40a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc005d00 tx=0x7ff6cc005c50 comp rx=0 tx=0).stop 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 msgr2=0x7ff6dc075260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc075260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40097b0 tx=0x7ff6d4031840 comp rx=0 tx=0).stop 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 shutdown_connections 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff6c40a3bb0 0x7ff6c40a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6dc104540 0x7ff6dc075260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6dc103b80 0x7ff6dc074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 --2- 172.21.15.1:0/123613449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6dc102990 0x7ff6dc0766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.490+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 >> 172.21.15.1:0/123613449 conn(0x7ff6dc0fdf20 msgr2=0x7ff6dc106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.491+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 shutdown_connections 2024-04-26T22:47:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:26.491+0000 7ff6e190a640 1 -- 172.21.15.1:0/123613449 wait complete. 2024-04-26T22:47:26.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:26 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/123613449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:26.736 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:26 smithi186 systemd[1]: Starting Ceph mds.i for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:26.793 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:26 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/123613449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:26.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:26 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/123613449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:27.125 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:26 smithi186 podman[63159]: 2024-04-26 22:47:26.749290393 +0000 UTC m=+0.018067640 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:27.125 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:26 smithi186 podman[63159]: 2024-04-26 22:47:26.865093396 +0000 UTC m=+0.133870675 container create f399c051c443ab21bf378b1a82616a8793e674a2bf3e276da1d48437acfcb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T22:47:27.126 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:27 smithi186 podman[63159]: 2024-04-26 22:47:27.126099006 +0000 UTC m=+0.394876239 container init f399c051c443ab21bf378b1a82616a8793e674a2bf3e276da1d48437acfcb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-26T22:47:27.485 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:27 smithi186 podman[63159]: 2024-04-26 22:47:27.128802048 +0000 UTC m=+0.397579292 container start f399c051c443ab21bf378b1a82616a8793e674a2bf3e276da1d48437acfcb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T22:47:27.485 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mds[63178]: -- 172.21.15.186:0/3312518781 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d15ae83a40 con 0x55d15ae56c00 2024-04-26T22:47:27.485 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: starting mds.i at 2024-04-26T22:47:27.485 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:27 smithi186 bash[63159]: f399c051c443ab21bf378b1a82616a8793e674a2bf3e276da1d48437acfcb024 2024-04-26T22:47:27.485 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:27 smithi186 systemd[1]: Started Ceph mds.i for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:27 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:27 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:27 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:28.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 -- 172.21.15.1:0/816152902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed41045f0 msgr2=0x7f4ed410ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/816152902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed41045f0 0x7f4ed410ae80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009cd0 tx=0x7f4ecc02a8e0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 -- 172.21.15.1:0/816152902 shutdown_connections 2024-04-26T22:47:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/816152902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed4103c30 0x7f4ed41040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/816152902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed41045f0 0x7f4ed410ae80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/816152902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed4100c60 0x7f4ed4101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 -- 172.21.15.1:0/816152902 >> 172.21.15.1:0/816152902 conn(0x7f4ed40fc590 msgr2=0x7f4ed40fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 -- 172.21.15.1:0/816152902 shutdown_connections 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.614+0000 7f4ed8f40640 1 -- 172.21.15.1:0/816152902 wait complete. 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 Processor -- start 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 -- start start 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed4100c60 0x7f4ed4198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 0x7f4ed41990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed41045f0 0x7f4ed41a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410e4c0 con 0x7f4ed4103c30 2024-04-26T22:47:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410e340 con 0x7f4ed41045f0 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed8f40640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410e640 con 0x7f4ed4100c60 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 0x7f4ed41990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 0x7f4ed41990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48436/0 (socket says 172.21.15.1:48436) 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 -- 172.21.15.1:0/2596963611 learned_addr learned my addr 172.21.15.1:0/2596963611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed259c640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed4100c60 0x7f4ed4198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed2d9d640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed41045f0 0x7f4ed41a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 -- 172.21.15.1:0/2596963611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed4100c60 msgr2=0x7f4ed4198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed4100c60 0x7f4ed4198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 -- 172.21.15.1:0/2596963611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed41045f0 msgr2=0x7f4ed41a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed41045f0 0x7f4ed41a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.615+0000 7f4ed1d9b640 1 -- 172.21.15.1:0/2596963611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed41a0780 con 0x7f4ed4103c30 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.616+0000 7f4ed1d9b640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 0x7f4ed41990a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400d950 tx=0x7f4ec400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.616+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4017030 con 0x7f4ed4103c30 2024-04-26T22:47:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.616+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec4004020 con 0x7f4ed4103c30 2024-04-26T22:47:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.616+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed41a0a70 con 0x7f4ed4103c30 2024-04-26T22:47:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.616+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40047a0 con 0x7f4ed4103c30 2024-04-26T22:47:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.616+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed41a0fc0 con 0x7f4ed4103c30 2024-04-26T22:47:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.617+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed4109de0 con 0x7f4ed4103c30 2024-04-26T22:47:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.618+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4ec4009a30 con 0x7f4ed4103c30 2024-04-26T22:47:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.618+0000 7f4ec37fe640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4ebc0a3bb0 0x7f4ebc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.619+0000 7f4ed259c640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4ebc0a3bb0 0x7f4ebc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.619+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4ec40cf9d0 con 0x7f4ed4103c30 2024-04-26T22:47:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.619+0000 7f4ed259c640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4ebc0a3bb0 0x7f4ebc0a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4ed406ae40 tx=0x7f4ec800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.623+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4ec409c850 con 0x7f4ed4103c30 2024-04-26T22:47:28.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:28 smithi001 ceph-mon[29435]: Deploying daemon mds.g on smithi001 2024-04-26T22:47:28.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:28 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] up:boot 2024-04-26T22:47:28.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:28 smithi001 ceph-mon[29435]: fsmap 9 up:standby 2024-04-26T22:47:28.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:28 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T22:47:28.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.848+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ed402dad0 con 0x7f4ed4103c30 2024-04-26T22:47:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.849+0000 7f4ec37fe640 1 -- 172.21.15.1:0/2596963611 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7f4ec409c150 con 0x7f4ed4103c30 2024-04-26T22:47:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:28.850 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T22:47:28.851 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 10 2024-04-26T22:47:28.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:28 smithi038 ceph-mon[32433]: Deploying daemon mds.g on smithi001 2024-04-26T22:47:28.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:28 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] up:boot 2024-04-26T22:47:28.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:28 smithi038 ceph-mon[32433]: fsmap 9 up:standby 2024-04-26T22:47:28.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:28 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T22:47:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4ebc0a3bb0 msgr2=0x7f4ebc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4ebc0a3bb0 0x7f4ebc0a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4ed406ae40 tx=0x7f4ec800b390 comp rx=0 tx=0).stop 2024-04-26T22:47:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 msgr2=0x7f4ed41990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 0x7f4ed41990a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400d950 tx=0x7f4ec400de10 comp rx=0 tx=0).stop 2024-04-26T22:47:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 shutdown_connections 2024-04-26T22:47:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4ebc0a3bb0 0x7f4ebc0a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed41045f0 0x7f4ed41a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed4103c30 0x7f4ed41990a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 --2- 172.21.15.1:0/2596963611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed4100c60 0x7f4ed4198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 >> 172.21.15.1:0/2596963611 conn(0x7f4ed40fc590 msgr2=0x7f4ed4106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 shutdown_connections 2024-04-26T22:47:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:28.851+0000 7f4ed8f40640 1 -- 172.21.15.1:0/2596963611 wait complete. 2024-04-26T22:47:28.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:28 smithi186 ceph-mon[32774]: Deploying daemon mds.g on smithi001 2024-04-26T22:47:28.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:28 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] up:boot 2024-04-26T22:47:28.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:28 smithi186 ceph-mon[32774]: fsmap 9 up:standby 2024-04-26T22:47:28.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:28 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T22:47:29.555 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:29 smithi001 systemd[1]: Starting Ceph mds.g for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:29 smithi001 ceph-mon[29435]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:29 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2596963611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:29.846 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:29 smithi001 podman[69099]: 2024-04-26 22:47:29.596379322 +0000 UTC m=+0.018578754 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:29.846 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:29 smithi001 podman[69099]: 2024-04-26 22:47:29.717825503 +0000 UTC m=+0.140024916 container create e60acb5583e374d30443fbbdb4a4ac6e3225a22213ca60634a4a712113a35b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T22:47:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:29 smithi038 ceph-mon[32433]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:29 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2596963611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:29.886 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:29 smithi186 ceph-mon[32774]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:29.886 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:29 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2596963611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:30.346 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:29 smithi001 podman[69099]: 2024-04-26 22:47:29.996159722 +0000 UTC m=+0.418359133 container init e60acb5583e374d30443fbbdb4a4ac6e3225a22213ca60634a4a712113a35b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T22:47:30.346 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:29 smithi001 podman[69099]: 2024-04-26 22:47:29.998816371 +0000 UTC m=+0.421015790 container start e60acb5583e374d30443fbbdb4a4ac6e3225a22213ca60634a4a712113a35b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T22:47:30.346 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:30 smithi001 ceph-mds[69118]: -- 172.21.15.1:0/539251778 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c8d889a40 con 0x560c8d85d000 2024-04-26T22:47:30.346 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:30 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g[69114]: starting mds.g at 2024-04-26T22:47:30.347 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:30 smithi001 bash[69099]: e60acb5583e374d30443fbbdb4a4ac6e3225a22213ca60634a4a712113a35b95 2024-04-26T22:47:30.347 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 22:47:30 smithi001 systemd[1]: Started Ceph mds.g for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:30.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 -- 172.21.15.1:0/4233592242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 msgr2=0x7f183010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 --2- 172.21.15.1:0/4233592242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f183010add0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1828009ca0 tx=0x7f182802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 -- 172.21.15.1:0/4233592242 shutdown_connections 2024-04-26T22:47:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 --2- 172.21.15.1:0/4233592242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f183010add0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 --2- 172.21.15.1:0/4233592242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1830103b80 0x7f1830104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 --2- 172.21.15.1:0/4233592242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830102990 0x7f1830102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 -- 172.21.15.1:0/4233592242 >> 172.21.15.1:0/4233592242 conn(0x7f18300fdf20 msgr2=0x7f1830100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 -- 172.21.15.1:0/4233592242 shutdown_connections 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 -- 172.21.15.1:0/4233592242 wait complete. 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 Processor -- start 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.975+0000 7f18358f1640 1 -- start start 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830102990 0x7f18300766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1830103b80 0x7f1830074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f1830075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f183010e4a0 con 0x7f1830104540 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f183010e320 con 0x7f1830103b80 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f183010e620 con 0x7f1830102990 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f1830075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f1830075260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48468/0 (socket says 172.21.15.1:48468) 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 -- 172.21.15.1:0/1652630037 learned_addr learned my addr 172.21.15.1:0/1652630037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182effd640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830102990 0x7f18300766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18267fc640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1830103b80 0x7f1830074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 -- 172.21.15.1:0/1652630037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830102990 msgr2=0x7f18300766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830102990 0x7f18300766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 -- 172.21.15.1:0/1652630037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1830103b80 msgr2=0x7f1830074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1830103b80 0x7f1830074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 -- 172.21.15.1:0/1652630037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1830075af0 con 0x7f1830104540 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f182f7fe640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f1830075260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f18280097b0 tx=0x7f1828031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828003bf0 con 0x7f1830104540 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18301a2c70 con 0x7f1830104540 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1828003d90 con 0x7f1830104540 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.976+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182802fc70 con 0x7f1830104540 2024-04-26T22:47:30.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.977+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18301a31c0 con 0x7f1830104540 2024-04-26T22:47:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.978+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1830109d30 con 0x7f1830104540 2024-04-26T22:47:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.979+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1828004b40 con 0x7f1830104540 2024-04-26T22:47:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.980+0000 7f18348ef640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f181c0a3bb0 0x7f181c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.980+0000 7f182effd640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f181c0a3bb0 0x7f181c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.980+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f18280f16a0 con 0x7f1830104540 2024-04-26T22:47:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.980+0000 7f182effd640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f181c0a3bb0 0x7f181c0a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f18200095f0 tx=0x7f1820005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:30.984+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f18280be520 con 0x7f1830104540 2024-04-26T22:47:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.216+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f183002dad0 con 0x7f1830104540 2024-04-26T22:47:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.216+0000 7f18348ef640 1 -- 172.21.15.1:0/1652630037 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7f18280bde20 con 0x7f1830104540 2024-04-26T22:47:31.218 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:31.218 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T22:47:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 11 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f181c0a3bb0 msgr2=0x7f181c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f181c0a3bb0 0x7f181c0a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f18200095f0 tx=0x7f1820005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 msgr2=0x7f1830075260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f1830075260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f18280097b0 tx=0x7f1828031840 comp rx=0 tx=0).stop 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 shutdown_connections 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f181c0a3bb0 0x7f181c0a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1830104540 0x7f1830075260 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1830103b80 0x7f1830074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 --2- 172.21.15.1:0/1652630037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830102990 0x7f18300766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 >> 172.21.15.1:0/1652630037 conn(0x7f18300fdf20 msgr2=0x7f1830106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 shutdown_connections 2024-04-26T22:47:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:31.218+0000 7f18358f1640 1 -- 172.21.15.1:0/1652630037 wait complete. 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: Deploying daemon mds.a on smithi001 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6838/1512535471,v1:172.21.15.1:6839/1512535471] up:boot 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: fsmap 10 up:standby 2024-04-26T22:47:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:31 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T22:47:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:31.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:31.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:31.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: Deploying daemon mds.a on smithi001 2024-04-26T22:47:31.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6838/1512535471,v1:172.21.15.1:6839/1512535471] up:boot 2024-04-26T22:47:31.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: fsmap 10 up:standby 2024-04-26T22:47:31.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:31 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T22:47:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: Deploying daemon mds.a on smithi001 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6838/1512535471,v1:172.21.15.1:6839/1512535471] up:boot 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: fsmap 10 up:standby 2024-04-26T22:47:31.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:31 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T22:47:32.301 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:32 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1652630037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:32.301 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 systemd[1]: Starting Ceph mds.a for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T22:47:32.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:32 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1652630037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:32.596 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 podman[69443]: 2024-04-26 22:47:32.507599883 +0000 UTC m=+0.103632583 container create 06e44256bb9331e0e3563df61fb9fd96508d4f91c0d6663c7c92c3b198438356 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T22:47:32.596 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 podman[69443]: 2024-04-26 22:47:32.422082878 +0000 UTC m=+0.018115563 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T22:47:32.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:32 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1652630037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:33.096 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 podman[69443]: 2024-04-26 22:47:32.735114013 +0000 UTC m=+0.331146679 container init 06e44256bb9331e0e3563df61fb9fd96508d4f91c0d6663c7c92c3b198438356 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T22:47:33.096 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 podman[69443]: 2024-04-26 22:47:32.737815992 +0000 UTC m=+0.333848659 container start 06e44256bb9331e0e3563df61fb9fd96508d4f91c0d6663c7c92c3b198438356 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T22:47:33.096 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 ceph-mds[69463]: -- 172.21.15.1:0/661115349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b741e9a40 con 0x556b741bc800 2024-04-26T22:47:33.096 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a[69459]: starting mds.a at 2024-04-26T22:47:33.096 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 bash[69443]: 06e44256bb9331e0e3563df61fb9fd96508d4f91c0d6663c7c92c3b198438356 2024-04-26T22:47:33.097 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 22:47:32 smithi001 systemd[1]: Started Ceph mds.a for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:33.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:33.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.340+0000 7f1ecedaf640 1 -- 172.21.15.1:0/3364811838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8105d40 msgr2=0x7f1ec8108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:33.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.340+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/3364811838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8105d40 0x7f1ec8108130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009900 tx=0x7f1ebc02a410 comp rx=0 tx=0).stop 2024-04-26T22:47:33.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- 172.21.15.1:0/3364811838 shutdown_connections 2024-04-26T22:47:33.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/3364811838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ec8108670 0x7f1ec810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/3364811838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8105d40 0x7f1ec8108130 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/3364811838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ec8103410 0x7f1ec8105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- 172.21.15.1:0/3364811838 >> 172.21.15.1:0/3364811838 conn(0x7f1ec80fce40 msgr2=0x7f1ec80ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- 172.21.15.1:0/3364811838 shutdown_connections 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- 172.21.15.1:0/3364811838 wait complete. 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 Processor -- start 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- start start 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 0x7f1ec819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ec8105d40 0x7f1ec819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ec8108670 0x7f1ec81a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec810e4b0 con 0x7f1ec8103410 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec810e330 con 0x7f1ec8108670 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1ecedaf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec810e630 con 0x7f1ec8105d40 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1eccb4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 0x7f1ec819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1eccb4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 0x7f1ec819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39818/0 (socket says 172.21.15.1:39818) 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.341+0000 7f1eccb4b640 1 -- 172.21.15.1:0/4054377817 learned_addr learned my addr 172.21.15.1:0/4054377817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ecd34c640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ec8108670 0x7f1ec81a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1eccb4b640 1 -- 172.21.15.1:0/4054377817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ec8105d40 msgr2=0x7f1ec819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ec7fff640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ec8105d40 0x7f1ec819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1eccb4b640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ec8105d40 0x7f1ec819b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1eccb4b640 1 -- 172.21.15.1:0/4054377817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ec8108670 msgr2=0x7f1ec81a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1eccb4b640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ec8108670 0x7f1ec81a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1eccb4b640 1 -- 172.21.15.1:0/4054377817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec81a2a30 con 0x7f1ec8103410 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1eccb4b640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 0x7f1ec819ae10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c890 tx=0x7f1eb800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8004220 con 0x7f1ec8103410 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1eb80043c0 con 0x7f1ec8103410 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec81a2d20 con 0x7f1ec8103410 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8004a00 con 0x7f1ec8103410 2024-04-26T22:47:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.342+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec81a31c0 con 0x7f1ec8103410 2024-04-26T22:47:33.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.343+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec810a1c0 con 0x7f1ec8103410 2024-04-26T22:47:33.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.344+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1eb80079b0 con 0x7f1ec8103410 2024-04-26T22:47:33.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.345+0000 7f1ec5ffb640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1eb00a3bb0 0x7f1eb00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:33.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.345+0000 7f1ec7fff640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1eb00a3bb0 0x7f1eb00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:33.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.345+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1eb80cfd40 con 0x7f1ec8103410 2024-04-26T22:47:33.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.345+0000 7f1ec7fff640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1eb00a3bb0 0x7f1eb00a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc031500 tx=0x7f1ebc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:33.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.349+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1eb8014030 con 0x7f1ec8103410 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:33 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:33.545 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:33.546 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.546 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.546 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.546 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:33.546 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:33.546 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:33 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.576+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ec8063560 con 0x7f1ec8103410 2024-04-26T22:47:33.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.577+0000 7f1ec5ffb640 1 -- 172.21.15.1:0/4054377817 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9644 (secure 0 0 0) 0x7f1eb809cb20 con 0x7f1ec8103410 2024-04-26T22:47:33.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:33.577 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T22:47:33.577 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 12 2024-04-26T22:47:33.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1eb00a3bb0 msgr2=0x7f1eb00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:33.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1eb00a3bb0 0x7f1eb00a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc031500 tx=0x7f1ebc000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 msgr2=0x7f1ec819ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 0x7f1ec819ae10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c890 tx=0x7f1eb800cd50 comp rx=0 tx=0).stop 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 shutdown_connections 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1eb00a3bb0 0x7f1eb00a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ec8105d40 0x7f1ec819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ec8108670 0x7f1ec81a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 --2- 172.21.15.1:0/4054377817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ec8103410 0x7f1ec819ae10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 >> 172.21.15.1:0/4054377817 conn(0x7f1ec80fce40 msgr2=0x7f1ec8102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 shutdown_connections 2024-04-26T22:47:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:33.579+0000 7f1ecedaf640 1 -- 172.21.15.1:0/4054377817 wait complete. 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T22:47:33.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:33 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:34 smithi186 ceph-mon[32774]: Deploying daemon mds.b on smithi038 2024-04-26T22:47:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:34 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] up:boot 2024-04-26T22:47:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:34 smithi186 ceph-mon[32774]: fsmap 11 up:standby 2024-04-26T22:47:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:34 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T22:47:34.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:34 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/4054377817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:34.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:34 smithi001 ceph-mon[29435]: Deploying daemon mds.b on smithi038 2024-04-26T22:47:34.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:34 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] up:boot 2024-04-26T22:47:34.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:34 smithi001 ceph-mon[29435]: fsmap 11 up:standby 2024-04-26T22:47:34.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:34 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T22:47:34.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4054377817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:34 smithi038 ceph-mon[32433]: Deploying daemon mds.b on smithi038 2024-04-26T22:47:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:34 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] up:boot 2024-04-26T22:47:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:34 smithi038 ceph-mon[32433]: fsmap 11 up:standby 2024-04-26T22:47:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:34 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T22:47:34.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:34 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/4054377817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:35 smithi038 ceph-mon[32433]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:35 smithi186 ceph-mon[32774]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:35.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:35 smithi001 ceph-mon[29435]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1946714688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 msgr2=0x7f6f5c105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1946714688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c105860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009900 tx=0x7f6f4c02a410 comp rx=0 tx=0).stop 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1946714688 shutdown_connections 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1946714688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f5c1086d0 0x7f6f5c10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1946714688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c105da0 0x7f6f5c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1946714688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c105860 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1946714688 >> 172.21.15.1:0/1946714688 conn(0x7f6f5c0fce00 msgr2=0x7f6f5c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1946714688 shutdown_connections 2024-04-26T22:47:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.706+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1946714688 wait complete. 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 Processor -- start 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 -- start start 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f5c105da0 0x7f6f5c1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c1086d0 0x7f6f5c1a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c10e4b0 con 0x7f6f5c103470 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c10e330 con 0x7f6f5c1086d0 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f61ff0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c10e630 con 0x7f6f5c105da0 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f5b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f5b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39838/0 (socket says 172.21.15.1:39838) 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f5b7fe640 1 -- 172.21.15.1:0/1056541271 learned_addr learned my addr 172.21.15.1:0/1056541271 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.707+0000 7f6f5bfff640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c1086d0 0x7f6f5c1a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5affd640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f5c105da0 0x7f6f5c1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5b7fe640 1 -- 172.21.15.1:0/1056541271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f5c105da0 msgr2=0x7f6f5c1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5b7fe640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f5c105da0 0x7f6f5c1990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5b7fe640 1 -- 172.21.15.1:0/1056541271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c1086d0 msgr2=0x7f6f5c1a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5b7fe640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c1086d0 0x7f6f5c1a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5b7fe640 1 -- 172.21.15.1:0/1056541271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c1a07a0 con 0x7f6f5c103470 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f5b7fe640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c198b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c005940 tx=0x7f6f4c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c03f030 con 0x7f6f5c103470 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5c1a0a30 con 0x7f6f5c103470 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f4c003aa0 con 0x7f6f5c103470 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c035070 con 0x7f6f5c103470 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.708+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5c1a0f20 con 0x7f6f5c103470 2024-04-26T22:47:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.709+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f5c0fe8f0 con 0x7f6f5c103470 2024-04-26T22:47:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.710+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6f4c042030 con 0x7f6f5c103470 2024-04-26T22:47:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.711+0000 7f6f58ff9640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6f440a3bb0 0x7f6f440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.711+0000 7f6f5affd640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6f440a3bb0 0x7f6f440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.711+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6f4c0f0640 con 0x7f6f5c103470 2024-04-26T22:47:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.711+0000 7f6f5affd640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6f440a3bb0 0x7f6f440a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c19a090 tx=0x7f6f50005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:35.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.715+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6f4c0bd420 con 0x7f6f5c103470 2024-04-26T22:47:35.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.957+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f5c0635d0 con 0x7f6f5c103470 2024-04-26T22:47:35.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.958+0000 7f6f58ff9640 1 -- 172.21.15.1:0/1056541271 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9644 (secure 0 0 0) 0x7f6f4c0bcd20 con 0x7f6f5c103470 2024-04-26T22:47:35.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:35.958 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T22:47:35.958 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 12 2024-04-26T22:47:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6f440a3bb0 msgr2=0x7f6f440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6f440a3bb0 0x7f6f440a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c19a090 tx=0x7f6f50005cc0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 msgr2=0x7f6f5c198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c198b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c005940 tx=0x7f6f4c031a50 comp rx=0 tx=0).stop 2024-04-26T22:47:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 shutdown_connections 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6f440a3bb0 0x7f6f440a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f5c105da0 0x7f6f5c1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c1086d0 0x7f6f5c1a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 --2- 172.21.15.1:0/1056541271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f5c103470 0x7f6f5c198b80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 >> 172.21.15.1:0/1056541271 conn(0x7f6f5c0fce00 msgr2=0x7f6f5c103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 shutdown_connections 2024-04-26T22:47:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:35.960+0000 7f6f61ff0640 1 -- 172.21.15.1:0/1056541271 wait complete. 2024-04-26T22:47:36.101 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 22:47:35 smithi038 systemd[1]: Started Ceph mds.b for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T22:47:37.074 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:37.074 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1056541271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6838/2490292790,v1:172.21.15.38:6839/2490292790] up:boot 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: fsmap 12 up:standby 2024-04-26T22:47:37.075 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:36 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1056541271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6838/2490292790,v1:172.21.15.38:6839/2490292790] up:boot 2024-04-26T22:47:37.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: fsmap 12 up:standby 2024-04-26T22:47:37.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:36 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1056541271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6838/2490292790,v1:172.21.15.38:6839/2490292790] up:boot 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: fsmap 12 up:standby 2024-04-26T22:47:37.131 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:36 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T22:47:37.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 -- 172.21.15.1:0/3024429709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 msgr2=0x7f81501016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 --2- 172.21.15.1:0/3024429709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f81501016a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8144009900 tx=0x7f814402a480 comp rx=0 tx=0).stop 2024-04-26T22:47:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 -- 172.21.15.1:0/3024429709 shutdown_connections 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 --2- 172.21.15.1:0/3024429709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8150102580 0x7f8150102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 --2- 172.21.15.1:0/3024429709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 0x7f8150102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 --2- 172.21.15.1:0/3024429709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f81501016a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 -- 172.21.15.1:0/3024429709 >> 172.21.15.1:0/3024429709 conn(0x7f81500fce00 msgr2=0x7f81500ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 -- 172.21.15.1:0/3024429709 shutdown_connections 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 -- 172.21.15.1:0/3024429709 wait complete. 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.077+0000 7f8156d6a640 1 Processor -- start 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 -- start start 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f815019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 0x7f815019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8150102580 0x7f81501a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f81501106e0 con 0x7f8150101280 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8150110560 con 0x7f8150101be0 2024-04-26T22:47:38.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8156d6a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8150110860 con 0x7f8150102580 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f815019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f815019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39852/0 (socket says 172.21.15.1:39852) 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 -- 172.21.15.1:0/888021083 learned_addr learned my addr 172.21.15.1:0/888021083 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8155307640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8150102580 0x7f81501a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f814ffff640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 0x7f815019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 -- 172.21.15.1:0/888021083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8150102580 msgr2=0x7f81501a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8150102580 0x7f81501a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 -- 172.21.15.1:0/888021083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 msgr2=0x7f815019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 0x7f815019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 -- 172.21.15.1:0/888021083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81501a29d0 con 0x7f8150101280 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f814ffff640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 0x7f815019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f8154b06640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f815019adb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8144005ab0 tx=0x7f8144003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.078+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814403f070 con 0x7f8150101280 2024-04-26T22:47:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.079+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8144031c50 con 0x7f8150101280 2024-04-26T22:47:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.079+0000 7f8156d6a640 1 -- 172.21.15.1:0/888021083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81501a2c60 con 0x7f8150101280 2024-04-26T22:47:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.079+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8144035030 con 0x7f8150101280 2024-04-26T22:47:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.079+0000 7f8156d6a640 1 -- 172.21.15.1:0/888021083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81501a3150 con 0x7f8150101280 2024-04-26T22:47:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.080+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8150063530 con 0x7f8150101280 2024-04-26T22:47:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.081+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8144042070 con 0x7f8150101280 2024-04-26T22:47:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.082+0000 7f814dffb640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f81400a3bb0 0x7f81400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.082+0000 7f814ffff640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f81400a3bb0 0x7f81400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.082+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f81440f0e60 con 0x7f8150101280 2024-04-26T22:47:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.082+0000 7f814ffff640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f81400a3bb0 0x7f81400a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f815019c2c0 tx=0x7f813c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.090+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f81440bd5f0 con 0x7f8150101280 2024-04-26T22:47:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.317+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8150063820 con 0x7f8150101280 2024-04-26T22:47:38.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.318+0000 7f814dffb640 1 -- 172.21.15.1:0/888021083 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10470 (secure 0 0 0) 0x7f81440c18f0 con 0x7f8150101280 2024-04-26T22:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T22:47:38.319 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 13 2024-04-26T22:47:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.320+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f81400a3bb0 msgr2=0x7f81400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.320+0000 7f813b7fe640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f81400a3bb0 0x7f81400a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f815019c2c0 tx=0x7f813c005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 msgr2=0x7f815019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f815019adb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8144005ab0 tx=0x7f8144003b90 comp rx=0 tx=0).stop 2024-04-26T22:47:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 shutdown_connections 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f81400a3bb0 0x7f81400a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8150102580 0x7f81501a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8150101be0 0x7f815019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 --2- 172.21.15.1:0/888021083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150101280 0x7f815019adb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 >> 172.21.15.1:0/888021083 conn(0x7f81500fce00 msgr2=0x7f81500fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 shutdown_connections 2024-04-26T22:47:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.321+0000 7f813b7fe640 1 -- 172.21.15.1:0/888021083 wait complete. 2024-04-26T22:47:38.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:47:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 -- 172.21.15.1:0/7369012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 msgr2=0x7f1340109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 --2- 172.21.15.1:0/7369012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f1340109cd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1338009ca0 tx=0x7f133802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 -- 172.21.15.1:0/7369012 shutdown_connections 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 --2- 172.21.15.1:0/7369012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f1340109cd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 --2- 172.21.15.1:0/7369012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1340102a80 0x7f1340102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 --2- 172.21.15.1:0/7369012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1340101890 0x7f1340101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 -- 172.21.15.1:0/7369012 >> 172.21.15.1:0/7369012 conn(0x7f13400fce00 msgr2=0x7f13400ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 -- 172.21.15.1:0/7369012 shutdown_connections 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.438+0000 7f1345a59640 1 -- 172.21.15.1:0/7369012 wait complete. 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 Processor -- start 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 -- start start 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1340101890 0x7f1340105820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1340102a80 0x7f1340103e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f13401043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f134010d3a0 con 0x7f1340103440 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f134010d220 con 0x7f1340102a80 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f1345a59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f134010d520 con 0x7f1340101890 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f133f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f13401043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f133f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f13401043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39858/0 (socket says 172.21.15.1:39858) 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f133f7fe640 1 -- 172.21.15.1:0/3947128105 learned_addr learned my addr 172.21.15.1:0/3947128105 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f133e7fc640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1340102a80 0x7f1340103e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.439+0000 7f133effd640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1340101890 0x7f1340105820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f133f7fe640 1 -- 172.21.15.1:0/3947128105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1340101890 msgr2=0x7f1340105820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f133f7fe640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1340101890 0x7f1340105820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f133f7fe640 1 -- 172.21.15.1:0/3947128105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1340102a80 msgr2=0x7f1340103e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f133f7fe640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1340102a80 0x7f1340103e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f133f7fe640 1 -- 172.21.15.1:0/3947128105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1340104c30 con 0x7f1340103440 2024-04-26T22:47:38.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f133f7fe640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f13401043a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f13380097b0 tx=0x7f1338031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:38.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338003b70 con 0x7f1340103440 2024-04-26T22:47:38.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13401a2c30 con 0x7f1340103440 2024-04-26T22:47:38.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1338003d10 con 0x7f1340103440 2024-04-26T22:47:38.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133802fc10 con 0x7f1340103440 2024-04-26T22:47:38.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.440+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13401a31d0 con 0x7f1340103440 2024-04-26T22:47:38.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.441+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13401a33a0 con 0x7f1340103440 2024-04-26T22:47:38.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.443+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1338004b40 con 0x7f1340103440 2024-04-26T22:47:38.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.443+0000 7f1344a57640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13280a3bb0 0x7f13280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.443+0000 7f133effd640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13280a3bb0 0x7f13280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.443+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f13380f15b0 con 0x7f1340103440 2024-04-26T22:47:38.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.444+0000 7f133effd640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13280a3bb0 0x7f13280a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f13300095f0 tx=0x7f1330005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:38.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.447+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f13380be430 con 0x7f1340103440 2024-04-26T22:47:38.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.656+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13401a3690 con 0x7f1340103440 2024-04-26T22:47:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.657+0000 7f1344a57640 1 -- 172.21.15.1:0/3947128105 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18277 (secure 0 0 0) 0x7f13380bdd30 con 0x7f1340103440 2024-04-26T22:47:38.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:38.657 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:46:49.379505+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:47:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13280a3bb0 msgr2=0x7f13280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13280a3bb0 0x7f13280a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f13300095f0 tx=0x7f1330005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 msgr2=0x7f13401043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f13401043a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f13380097b0 tx=0x7f1338031770 comp rx=0 tx=0).stop 2024-04-26T22:47:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 shutdown_connections 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13280a3bb0 0x7f13280a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1340103440 0x7f13401043a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1340102a80 0x7f1340103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 --2- 172.21.15.1:0/3947128105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1340101890 0x7f1340105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 >> 172.21.15.1:0/3947128105 conn(0x7f13400fce00 msgr2=0x7f13400fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 shutdown_connections 2024-04-26T22:47:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.659+0000 7f1345a59640 1 -- 172.21.15.1:0/3947128105 wait complete. 2024-04-26T22:47:38.670 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:47:38.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.775+0000 7f132ead9640 1 -- 172.21.15.1:0/3812705625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 msgr2=0x7f1328101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.775+0000 7f132ead9640 1 --2- 172.21.15.1:0/3812705625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328101070 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1320009cd0 tx=0x7f132002a8e0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- 172.21.15.1:0/3812705625 shutdown_connections 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 --2- 172.21.15.1:0/3812705625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1328103c20 0x7f13281040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 --2- 172.21.15.1:0/3812705625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13281045e0 0x7f132810ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 --2- 172.21.15.1:0/3812705625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328101070 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- 172.21.15.1:0/3812705625 >> 172.21.15.1:0/3812705625 conn(0x7f13280fc560 msgr2=0x7f13280fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- 172.21.15.1:0/3812705625 shutdown_connections 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- 172.21.15.1:0/3812705625 wait complete. 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 Processor -- start 2024-04-26T22:47:38.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- start start 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1328103c20 0x7f1328199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13281045e0 0x7f13281a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f132810e4d0 con 0x7f1328100c50 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f132810e350 con 0x7f13281045e0 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132ead9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f132810e650 con 0x7f1328103c20 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132c875640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132c875640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39876/0 (socket says 172.21.15.1:39876) 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132c875640 1 -- 172.21.15.1:0/2460321907 learned_addr learned my addr 172.21.15.1:0/2460321907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f132d076640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13281045e0 0x7f13281a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.776+0000 7f1327fff640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1328103c20 0x7f1328199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132c875640 1 -- 172.21.15.1:0/2460321907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1328103c20 msgr2=0x7f1328199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132c875640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1328103c20 0x7f1328199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132c875640 1 -- 172.21.15.1:0/2460321907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13281045e0 msgr2=0x7f13281a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132c875640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13281045e0 0x7f13281a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132c875640 1 -- 172.21.15.1:0/2460321907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13281a0850 con 0x7f1328100c50 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132c875640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328198c40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1320005e30 tx=0x7f13200093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320004040 con 0x7f1328100c50 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13281a0ae0 con 0x7f1328100c50 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1320004690 con 0x7f1328100c50 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320030ea0 con 0x7f1328100c50 2024-04-26T22:47:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.777+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13281a1080 con 0x7f1328100c50 2024-04-26T22:47:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.778+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1328109e70 con 0x7f1328100c50 2024-04-26T22:47:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.779+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1320030630 con 0x7f1328100c50 2024-04-26T22:47:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.780+0000 7f1325ffb640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13100a3bb0 0x7f13100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.780+0000 7f1327fff640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13100a3bb0 0x7f13100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.780+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f132002e030 con 0x7f1328100c50 2024-04-26T22:47:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.781+0000 7f1327fff640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13100a3bb0 0x7f13100a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f131c009bd0 tx=0x7f131c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:38.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.785+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f13200bdee0 con 0x7f1328100c50 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:38.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:38 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/888021083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:38 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/888021083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:38.977 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:38.978 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:38 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/888021083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.979+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1328063530 con 0x7f1328100c50 2024-04-26T22:47:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.979+0000 7f1325ffb640 1 -- 172.21.15.1:0/2460321907 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f13200bd7e0 con 0x7f1328100c50 2024-04-26T22:47:38.980 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13100a3bb0 msgr2=0x7f13100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13100a3bb0 0x7f13100a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f131c009bd0 tx=0x7f131c005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 msgr2=0x7f1328198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328198c40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1320005e30 tx=0x7f13200093f0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 shutdown_connections 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13100a3bb0 0x7f13100a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1328103c20 0x7f1328199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13281045e0 0x7f13281a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 --2- 172.21.15.1:0/2460321907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1328100c50 0x7f1328198c40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.981+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 >> 172.21.15.1:0/2460321907 conn(0x7f13280fc560 msgr2=0x7f1328107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.982+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 shutdown_connections 2024-04-26T22:47:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:38.982+0000 7f132ead9640 1 -- 172.21.15.1:0/2460321907 wait complete. 2024-04-26T22:47:38.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 -- 172.21.15.1:0/1788730999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 msgr2=0x7ffaf0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 --2- 172.21.15.1:0/1788730999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf0108130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009900 tx=0x7ffae402a410 comp rx=0 tx=0).stop 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 -- 172.21.15.1:0/1788730999 shutdown_connections 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 --2- 172.21.15.1:0/1788730999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0108670 0x7ffaf010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 --2- 172.21.15.1:0/1788730999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf0108130 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 --2- 172.21.15.1:0/1788730999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf0103410 0x7ffaf0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 -- 172.21.15.1:0/1788730999 >> 172.21.15.1:0/1788730999 conn(0x7ffaf00fce40 msgr2=0x7ffaf00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 -- 172.21.15.1:0/1788730999 shutdown_connections 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 -- 172.21.15.1:0/1788730999 wait complete. 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 Processor -- start 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.095+0000 7ffaf6319640 1 -- start start 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf6319640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0103410 0x7ffaf0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf6319640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf6319640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf0108670 0x7ffaf01a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf6319640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf010e510 con 0x7ffaf0105d40 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf6319640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf010e390 con 0x7ffaf0108670 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf6319640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf010e690 con 0x7ffaf0103410 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf01991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39894/0 (socket says 172.21.15.1:39894) 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 -- 172.21.15.1:0/517896070 learned_addr learned my addr 172.21.15.1:0/517896070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaf48b6640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf0108670 0x7ffaf01a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaeffff640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0103410 0x7ffaf0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 -- 172.21.15.1:0/517896070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0103410 msgr2=0x7ffaf0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0103410 0x7ffaf0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 -- 172.21.15.1:0/517896070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf0108670 msgr2=0x7ffaf01a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf0108670 0x7ffaf01a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 -- 172.21.15.1:0/517896070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf01a0870 con 0x7ffaf0105d40 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.096+0000 7ffaef7fe640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf01991a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ffae4006010 tx=0x7ffae40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.097+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae403f030 con 0x7ffaf0105d40 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.097+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffae4003cb0 con 0x7ffaf0105d40 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.097+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf01a0b00 con 0x7ffaf0105d40 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.097+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4035070 con 0x7ffaf0105d40 2024-04-26T22:47:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.097+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf01a1040 con 0x7ffaf0105d40 2024-04-26T22:47:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.098+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaf00fe930 con 0x7ffaf0105d40 2024-04-26T22:47:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.099+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ffae4042030 con 0x7ffaf0105d40 2024-04-26T22:47:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.100+0000 7ffaed7fa640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ffad80a3c30 0x7ffad80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.100+0000 7ffaeffff640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ffad80a3c30 0x7ffad80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.100+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffae40f04e0 con 0x7ffaf0105d40 2024-04-26T22:47:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.100+0000 7ffaeffff640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ffad80a3c30 0x7ffad80a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffaf019a130 tx=0x7ffae000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:39.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.104+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffae40bcd10 con 0x7ffaf0105d40 2024-04-26T22:47:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.291+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ffaf002dd20 con 0x7ffaf0105d40 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.291+0000 7ffaed7fa640 1 -- 172.21.15.1:0/517896070 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7ffae40dbe20 con 0x7ffaf0105d40 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-04-26T22:47:39.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T22:47:39.293 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.293+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ffad80a3c30 msgr2=0x7ffad80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.293+0000 7ffaf6319640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ffad80a3c30 0x7ffad80a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffaf019a130 tx=0x7ffae000b390 comp rx=0 tx=0).stop 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.293+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 msgr2=0x7ffaf01991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.293+0000 7ffaf6319640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf01991a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ffae4006010 tx=0x7ffae40042f0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 shutdown_connections 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ffad80a3c30 0x7ffad80a60f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf0108670 0x7ffaf01a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf0105d40 0x7ffaf01991a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 --2- 172.21.15.1:0/517896070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0103410 0x7ffaf0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 >> 172.21.15.1:0/517896070 conn(0x7ffaf00fce40 msgr2=0x7ffaf0103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:39.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 shutdown_connections 2024-04-26T22:47:39.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.294+0000 7ffaf6319640 1 -- 172.21.15.1:0/517896070 wait complete. 2024-04-26T22:47:39.306 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T22:47:39.306 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/1758714844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc108650 msgr2=0x7fa2dc10aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/1758714844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc108650 0x7fa2dc10aaa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4009ca0 tx=0x7fa2d402a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/1758714844 shutdown_connections 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/1758714844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc108650 0x7fa2dc10aaa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/1758714844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc105d20 0x7fa2dc108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/1758714844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2dc1033f0 0x7fa2dc1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/1758714844 >> 172.21.15.1:0/1758714844 conn(0x7fa2dc0fce00 msgr2=0x7fa2dc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/1758714844 shutdown_connections 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.419+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/1758714844 wait complete. 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 Processor -- start 2024-04-26T22:47:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 -- start start 2024-04-26T22:47:39.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2dc1033f0 0x7fa2dc198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 0x7fa2dc1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc108650 0x7fa2dc1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2dc10e4c0 con 0x7fa2dc105d20 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2dc10e340 con 0x7fa2dc108650 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2e1f2b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2dc10e640 con 0x7fa2dc1033f0 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2d2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 0x7fa2dc1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2d2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 0x7fa2dc1990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39900/0 (socket says 172.21.15.1:39900) 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2dbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc108650 0x7fa2dc1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.420+0000 7fa2d2ffd640 1 -- 172.21.15.1:0/3900207546 learned_addr learned my addr 172.21.15.1:0/3900207546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2db7fe640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2dc1033f0 0x7fa2dc198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d2ffd640 1 -- 172.21.15.1:0/3900207546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2dc1033f0 msgr2=0x7fa2dc198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d2ffd640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2dc1033f0 0x7fa2dc198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d2ffd640 1 -- 172.21.15.1:0/3900207546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc108650 msgr2=0x7fa2dc1a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d2ffd640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc108650 0x7fa2dc1a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d2ffd640 1 -- 172.21.15.1:0/3900207546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2dc1a07b0 con 0x7fa2dc105d20 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2dbfff640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc108650 0x7fa2dc1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d2ffd640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 0x7fa2dc1990d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400e9c0 tx=0x7fa2c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c400cd80 con 0x7fa2dc105d20 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c40054d0 con 0x7fa2dc105d20 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2dc1a0aa0 con 0x7fa2dc105d20 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4010430 con 0x7fa2dc105d20 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.421+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2dc1a0ff0 con 0x7fa2dc105d20 2024-04-26T22:47:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.422+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2dc0fe8f0 con 0x7fa2dc105d20 2024-04-26T22:47:39.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.424+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa2c4004fb0 con 0x7fa2dc105d20 2024-04-26T22:47:39.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.425+0000 7fa2d97fa640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.425+0000 7fa2db7fe640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.425+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa2c40d05e0 con 0x7fa2dc105d20 2024-04-26T22:47:39.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.425+0000 7fa2db7fe640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009bd0 tx=0x7fa2cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:39.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.429+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa2c4054060 con 0x7fa2dc105d20 2024-04-26T22:47:39.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.626+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa2dc02dad0 con 0x7fa2dc105d20 2024-04-26T22:47:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.748+0000 7fa2d97fa640 1 -- 172.21.15.1:0/3900207546 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fa2c409d3c0 con 0x7fa2dc105d20 2024-04-26T22:47:39.749 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-04-26T22:47:39.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa2c80a3bb0 msgr2=0x7fa2c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009bd0 tx=0x7fa2cc005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:39.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 msgr2=0x7fa2dc1990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 0x7fa2dc1990d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400e9c0 tx=0x7fa2c400ee80 comp rx=0 tx=0).stop 2024-04-26T22:47:39.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 shutdown_connections 2024-04-26T22:47:39.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2dc108650 0x7fa2dc1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2dc105d20 0x7fa2dc1990d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 --2- 172.21.15.1:0/3900207546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2dc1033f0 0x7fa2dc198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 >> 172.21.15.1:0/3900207546 conn(0x7fa2dc0fce00 msgr2=0x7fa2dc103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 shutdown_connections 2024-04-26T22:47:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.751+0000 7fa2e1f2b640 1 -- 172.21.15.1:0/3900207546 wait complete. 2024-04-26T22:47:39.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T22:47:39.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 -- 172.21.15.1:0/4085553101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 msgr2=0x7fcc38102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 --2- 172.21.15.1:0/4085553101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc38102f10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009900 tx=0x7fcc2002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 -- 172.21.15.1:0/4085553101 shutdown_connections 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 --2- 172.21.15.1:0/4085553101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 0x7fcc38109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 --2- 172.21.15.1:0/4085553101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc38102f10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 --2- 172.21.15.1:0/4085553101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 0x7fcc38101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 -- 172.21.15.1:0/4085553101 >> 172.21.15.1:0/4085553101 conn(0x7fcc380fce10 msgr2=0x7fcc380ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 -- 172.21.15.1:0/4085553101 shutdown_connections 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.877+0000 7fcc3f264640 1 -- 172.21.15.1:0/4085553101 wait complete. 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 Processor -- start 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 -- start start 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 0x7fcc3806bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc3819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 0x7fcc3819fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3810d3d0 con 0x7fcc38102a90 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3810d250 con 0x7fcc38103450 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3f264640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3810d550 con 0x7fcc381018a0 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc3819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc3819d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39910/0 (socket says 172.21.15.1:39910) 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 -- 172.21.15.1:0/2584713131 learned_addr learned my addr 172.21.15.1:0/2584713131 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3d801640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 0x7fcc3819fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3d000640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 0x7fcc3806bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 -- 172.21.15.1:0/2584713131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 msgr2=0x7fcc3806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 0x7fcc3806bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 -- 172.21.15.1:0/2584713131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 msgr2=0x7fcc3819fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 0x7fcc3819fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 -- 172.21.15.1:0/2584713131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc381a0790 con 0x7fcc38102a90 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3d801640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 0x7fcc3819fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:47:39.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc3d000640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 0x7fcc3806bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:39.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.878+0000 7fcc37fff640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc3819d860 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc20005940 tx=0x7fcc20004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:39.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.879+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2003f030 con 0x7fcc38102a90 2024-04-26T22:47:39.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.879+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc381a09c0 con 0x7fcc38102a90 2024-04-26T22:47:39.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.879+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc20003cb0 con 0x7fcc38102a90 2024-04-26T22:47:39.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.879+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc20035070 con 0x7fcc38102a90 2024-04-26T22:47:39.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.879+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc381a0f00 con 0x7fcc38102a90 2024-04-26T22:47:39.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.880+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc38108bc0 con 0x7fcc38102a90 2024-04-26T22:47:39.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.881+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fcc20042030 con 0x7fcc38102a90 2024-04-26T22:47:39.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.882+0000 7fcc35ffb640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcc240a3bb0 0x7fcc240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:39.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.882+0000 7fcc3d000640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcc240a3bb0 0x7fcc240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:39.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.882+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fcc200f0d50 con 0x7fcc38102a90 2024-04-26T22:47:39.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.882+0000 7fcc3d000640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcc240a3bb0 0x7fcc240a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcc381028f0 tx=0x7fcc28009400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:39.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:39.885+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcc200bda60 con 0x7fcc38102a90 2024-04-26T22:47:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:39 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3947128105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:39 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2460321907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:47:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:39 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/517896070' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T22:47:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:39 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3900207546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T22:47:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.077+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fcc38063530 con 0x7fcc38102a90 2024-04-26T22:47:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:39 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3947128105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:39 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2460321907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:47:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:39 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/517896070' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T22:47:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:39 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3900207546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T22:47:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:39 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3947128105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:39 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2460321907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:47:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:39 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/517896070' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T22:47:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:39 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3900207546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T22:47:40.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.751+0000 7fcc35ffb640 1 -- 172.21.15.1:0/2584713131 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7fcc20003e60 con 0x7fcc38102a90 2024-04-26T22:47:40.752 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-04-26T22:47:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcc240a3bb0 msgr2=0x7fcc240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcc240a3bb0 0x7fcc240a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcc381028f0 tx=0x7fcc28009400 comp rx=0 tx=0).stop 2024-04-26T22:47:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 msgr2=0x7fcc3819d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc3819d860 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc20005940 tx=0x7fcc20004430 comp rx=0 tx=0).stop 2024-04-26T22:47:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 shutdown_connections 2024-04-26T22:47:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcc240a3bb0 0x7fcc240a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc38103450 0x7fcc3819fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc38102a90 0x7fcc3819d860 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 --2- 172.21.15.1:0/2584713131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc381018a0 0x7fcc3806bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 >> 172.21.15.1:0/2584713131 conn(0x7fcc380fce10 msgr2=0x7fcc38105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 shutdown_connections 2024-04-26T22:47:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.756+0000 7fcc3f264640 1 -- 172.21.15.1:0/2584713131 wait complete. 2024-04-26T22:47:40.779 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.890+0000 7fae5487e640 1 -- 172.21.15.1:0/3224621496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50103430 msgr2=0x7fae50109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.890+0000 7fae5487e640 1 --2- 172.21.15.1:0/3224621496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50103430 0x7fae50109cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fae40009ca0 tx=0x7fae4002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 -- 172.21.15.1:0/3224621496 shutdown_connections 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 --2- 172.21.15.1:0/3224621496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50103430 0x7fae50109cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 --2- 172.21.15.1:0/3224621496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae50102a70 0x7fae50102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 --2- 172.21.15.1:0/3224621496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 0x7fae50101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 -- 172.21.15.1:0/3224621496 >> 172.21.15.1:0/3224621496 conn(0x7fae500fce10 msgr2=0x7fae500ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 -- 172.21.15.1:0/3224621496 shutdown_connections 2024-04-26T22:47:40.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 -- 172.21.15.1:0/3224621496 wait complete. 2024-04-26T22:47:40.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 Processor -- start 2024-04-26T22:47:40.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.891+0000 7fae5487e640 1 -- start start 2024-04-26T22:47:40.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 0x7fae50198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:40.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 0x7fae501990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:40.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae50103430 0x7fae501a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:40.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fae5010d390 con 0x7fae50102a70 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae5010d210 con 0x7fae50103430 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae5010d510 con 0x7fae50101880 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 0x7fae501990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 0x7fae501990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39928/0 (socket says 172.21.15.1:39928) 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 -- 172.21.15.1:0/376441185 learned_addr learned my addr 172.21.15.1:0/376441185 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4e59c640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 0x7fae50198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4ed9d640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae50103430 0x7fae501a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 -- 172.21.15.1:0/376441185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 msgr2=0x7fae50198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 0x7fae50198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 -- 172.21.15.1:0/376441185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae50103430 msgr2=0x7fae501a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae50103430 0x7fae501a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 -- 172.21.15.1:0/376441185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae501a0780 con 0x7fae50102a70 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4e59c640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 0x7fae50198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae4dd9b640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 0x7fae501990f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae3800d950 tx=0x7fae3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:40.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38017070 con 0x7fae50102a70 2024-04-26T22:47:40.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae38009c30 con 0x7fae50102a70 2024-04-26T22:47:40.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae501a0a70 con 0x7fae50102a70 2024-04-26T22:47:40.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.892+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3800f620 con 0x7fae50102a70 2024-04-26T22:47:40.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.893+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae501a0fc0 con 0x7fae50102a70 2024-04-26T22:47:40.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.893+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae50069c30 con 0x7fae50102a70 2024-04-26T22:47:40.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.895+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fae38010460 con 0x7fae50102a70 2024-04-26T22:47:40.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.895+0000 7fae477fe640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae3c0a3bb0 0x7fae3c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:40.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.895+0000 7fae4e59c640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae3c0a3bb0 0x7fae3c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:40.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.895+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fae380cfdb0 con 0x7fae50102a70 2024-04-26T22:47:40.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.896+0000 7fae4e59c640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae3c0a3bb0 0x7fae3c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fae501028d0 tx=0x7fae48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:40.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:40.899+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fae38009de0 con 0x7fae50102a70 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3900207546' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:40.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:40 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2584713131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T22:47:41.092 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.092 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3900207546' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.093 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:40 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2584713131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3900207546' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:47:41.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:40 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2584713131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T22:47:41.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.128+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fae501a11f0 con 0x7fae50102a70 2024-04-26T22:47:41.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.767+0000 7fae477fe640 1 -- 172.21.15.1:0/376441185 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fae3809c2f0 con 0x7fae50102a70 2024-04-26T22:47:41.767 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T22:47:41.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae3c0a3bb0 msgr2=0x7fae3c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:41.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae3c0a3bb0 0x7fae3c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fae501028d0 tx=0x7fae48005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:41.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 msgr2=0x7fae501990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 0x7fae501990f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae3800d950 tx=0x7fae3800de10 comp rx=0 tx=0).stop 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 shutdown_connections 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fae3c0a3bb0 0x7fae3c0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae50103430 0x7fae501a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae50102a70 0x7fae501990f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 --2- 172.21.15.1:0/376441185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae50101880 0x7fae50198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 >> 172.21.15.1:0/376441185 conn(0x7fae500fce10 msgr2=0x7fae500fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 shutdown_connections 2024-04-26T22:47:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.771+0000 7fae5487e640 1 -- 172.21.15.1:0/376441185 wait complete. 2024-04-26T22:47:41.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:47:41.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 -- 172.21.15.1:0/679778052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c1034d0 msgr2=0x7f567c1058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 --2- 172.21.15.1:0/679778052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c1034d0 0x7f567c1058c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f566c009900 tx=0x7f566c02a410 comp rx=0 tx=0).stop 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 -- 172.21.15.1:0/679778052 shutdown_connections 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 --2- 172.21.15.1:0/679778052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c108730 0x7f567c10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 --2- 172.21.15.1:0/679778052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f567c105e00 0x7f567c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 --2- 172.21.15.1:0/679778052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c1034d0 0x7f567c1058c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 -- 172.21.15.1:0/679778052 >> 172.21.15.1:0/679778052 conn(0x7f567c0fcea0 msgr2=0x7f567c0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 -- 172.21.15.1:0/679778052 shutdown_connections 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 -- 172.21.15.1:0/679778052 wait complete. 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 Processor -- start 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.897+0000 7f5682f14640 1 -- start start 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c1034d0 0x7f567c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 0x7f567c1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f567c108730 0x7f567c1a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f567c10e4a0 con 0x7f567c105e00 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f567c10e320 con 0x7f567c108730 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f567c10e620 con 0x7f567c1034d0 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 0x7f567c1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 0x7f567c1990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36580/0 (socket says 172.21.15.1:36580) 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 -- 172.21.15.1:0/3199307278 learned_addr learned my addr 172.21.15.1:0/3199307278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5680cb0640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c1034d0 0x7f567c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f56814b1640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f567c108730 0x7f567c1a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 -- 172.21.15.1:0/3199307278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c1034d0 msgr2=0x7f567c198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c1034d0 0x7f567c198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 -- 172.21.15.1:0/3199307278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f567c108730 msgr2=0x7f567c1a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f567c108730 0x7f567c1a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 -- 172.21.15.1:0/3199307278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c1a07c0 con 0x7f567c105e00 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5680cb0640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c1034d0 0x7f567c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f567bfff640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 0x7f567c1990e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5670005fd0 tx=0x7f567000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670017070 con 0x7f567c105e00 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5670009c30 con 0x7f567c105e00 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567c1a0ab0 con 0x7f567c105e00 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.898+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670003d30 con 0x7f567c105e00 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.899+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f567c1a1050 con 0x7f567c105e00 2024-04-26T22:47:41.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.899+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f567c0fe990 con 0x7f567c105e00 2024-04-26T22:47:41.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.900+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5670003ed0 con 0x7f567c105e00 2024-04-26T22:47:41.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.901+0000 7f5679ffb640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f56640a3c30 0x7f56640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:41.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.901+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f56700cffb0 con 0x7f567c105e00 2024-04-26T22:47:41.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.901+0000 7f5680cb0640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f56640a3c30 0x7f56640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:41.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.901+0000 7f5680cb0640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f56640a3c30 0x7f56640a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f566c006010 tx=0x7f566c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:41.905+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5670020030 con 0x7f567c105e00 2024-04-26T22:47:42.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.094+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f567c02dcf0 con 0x7f567c105e00 2024-04-26T22:47:42.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.095+0000 7f5679ffb640 1 -- 172.21.15.1:0/3199307278 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22053 (secure 0 0 0) 0x7f5670009de0 con 0x7f567c105e00 2024-04-26T22:47:42.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:42.096 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":84,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:47:41.130082+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:47:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:41 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2584713131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T22:47:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:41 smithi001 ceph-mon[29435]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T22:47:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:41 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/376441185' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T22:47:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:41 smithi001 ceph-mon[29435]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T22:47:42.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.097+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f56640a3c30 msgr2=0x7f56640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.097+0000 7f5682f14640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f56640a3c30 0x7f56640a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f566c006010 tx=0x7f566c000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.097+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 msgr2=0x7f567c1990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.097+0000 7f5682f14640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 0x7f567c1990e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5670005fd0 tx=0x7f567000dde0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 shutdown_connections 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f56640a3c30 0x7f56640a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f567c108730 0x7f567c1a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567c105e00 0x7f567c1990e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 --2- 172.21.15.1:0/3199307278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c1034d0 0x7f567c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 >> 172.21.15.1:0/3199307278 conn(0x7f567c0fcea0 msgr2=0x7f567c103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 shutdown_connections 2024-04-26T22:47:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.098+0000 7f5682f14640 1 -- 172.21.15.1:0/3199307278 wait complete. 2024-04-26T22:47:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:41 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2584713131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T22:47:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:41 smithi038 ceph-mon[32433]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T22:47:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:41 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/376441185' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T22:47:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:41 smithi038 ceph-mon[32433]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T22:47:42.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 -- 172.21.15.1:0/4259699229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc103410 msgr2=0x7f76bc105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/4259699229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc103410 0x7f76bc105800 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009900 tx=0x7f76ac02a410 comp rx=0 tx=0).stop 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 -- 172.21.15.1:0/4259699229 shutdown_connections 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/4259699229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc108670 0x7f76bc10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/4259699229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc105d40 0x7f76bc108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/4259699229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc103410 0x7f76bc105800 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 -- 172.21.15.1:0/4259699229 >> 172.21.15.1:0/4259699229 conn(0x7f76bc0fce40 msgr2=0x7f76bc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 -- 172.21.15.1:0/4259699229 shutdown_connections 2024-04-26T22:47:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.211+0000 7f76c0bc9640 1 -- 172.21.15.1:0/4259699229 wait complete. 2024-04-26T22:47:42.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 Processor -- start 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 -- start start 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc103410 0x7f76bc198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc105d40 0x7f76bc199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 0x7f76bc1a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc10e5b0 con 0x7f76bc108670 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc10e430 con 0x7f76bc103410 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76c0bc9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc10e730 con 0x7f76bc105d40 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76bad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 0x7f76bc1a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76bad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 0x7f76bc1a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36586/0 (socket says 172.21.15.1:36586) 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76bad9d640 1 -- 172.21.15.1:0/2531752841 learned_addr learned my addr 172.21.15.1:0/2531752841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.212+0000 7f76b9d9b640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc105d40 0x7f76bc199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76ba59c640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc103410 0x7f76bc198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76bad9d640 1 -- 172.21.15.1:0/2531752841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc105d40 msgr2=0x7f76bc199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76bad9d640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc105d40 0x7f76bc199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76bad9d640 1 -- 172.21.15.1:0/2531752841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc103410 msgr2=0x7f76bc198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76bad9d640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc103410 0x7f76bc198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76bad9d640 1 -- 172.21.15.1:0/2531752841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc1a0960 con 0x7f76bc108670 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76ba59c640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc103410 0x7f76bc198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76bad9d640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 0x7f76bc1a02f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b970 tx=0x7f76b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4019070 con 0x7f76bc108670 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc1a0c50 con 0x7f76bc108670 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc1a1220 con 0x7f76bc108670 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76b4014440 con 0x7f76bc108670 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.213+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b401d610 con 0x7f76bc108670 2024-04-26T22:47:42.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.214+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f769c0052c0 con 0x7f76bc108670 2024-04-26T22:47:42.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.215+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f76b401c070 con 0x7f76bc108670 2024-04-26T22:47:42.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.216+0000 7f76ab7fe640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f76a40a3bb0 0x7f76a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.216+0000 7f76ba59c640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f76a40a3bb0 0x7f76a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.216+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f76b40cf7a0 con 0x7f76bc108670 2024-04-26T22:47:42.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.216+0000 7f76ba59c640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f76a40a3bb0 0x7f76a40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f76ac006010 tx=0x7f76ac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:42.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.220+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f76b4014b10 con 0x7f76bc108670 2024-04-26T22:47:42.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:41 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2584713131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T22:47:42.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:41 smithi186 ceph-mon[32774]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T22:47:42.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:41 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/376441185' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T22:47:42.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:41 smithi186 ceph-mon[32774]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T22:47:42.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.411+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f769c005880 con 0x7f76bc108670 2024-04-26T22:47:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.412+0000 7f76ab7fe640 1 -- 172.21.15.1:0/2531752841 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22053 (secure 0 0 0) 0x7f76b409c2d0 con 0x7f76bc108670 2024-04-26T22:47:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:42.413 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":84,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:47:41.130082+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:47:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f76a40a3bb0 msgr2=0x7f76a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f76a40a3bb0 0x7f76a40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f76ac006010 tx=0x7f76ac000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 msgr2=0x7f76bc1a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 0x7f76bc1a02f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b970 tx=0x7f76b400bd70 comp rx=0 tx=0).stop 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 shutdown_connections 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f76a40a3bb0 0x7f76a40a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc105d40 0x7f76bc199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc108670 0x7f76bc1a02f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 --2- 172.21.15.1:0/2531752841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76bc103410 0x7f76bc198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 >> 172.21.15.1:0/2531752841 conn(0x7f76bc0fce40 msgr2=0x7f76bc103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 shutdown_connections 2024-04-26T22:47:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.415+0000 7f76c0bc9640 1 -- 172.21.15.1:0/2531752841 wait complete. 2024-04-26T22:47:42.427 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 -- 172.21.15.1:0/1716511758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34101870 msgr2=0x7f4c34101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 --2- 172.21.15.1:0/1716511758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34101870 0x7f4c34101c90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009900 tx=0x7f4c2402a410 comp rx=0 tx=0).stop 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 -- 172.21.15.1:0/1716511758 shutdown_connections 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 --2- 172.21.15.1:0/1716511758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34103420 0x7f4c34109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 --2- 172.21.15.1:0/1716511758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 0x7f4c34102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 --2- 172.21.15.1:0/1716511758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34101870 0x7f4c34101c90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.530+0000 7f4c39893640 1 -- 172.21.15.1:0/1716511758 >> 172.21.15.1:0/1716511758 conn(0x7f4c340fce00 msgr2=0x7f4c340ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 -- 172.21.15.1:0/1716511758 shutdown_connections 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 -- 172.21.15.1:0/1716511758 wait complete. 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 Processor -- start 2024-04-26T22:47:42.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 -- start start 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34101870 0x7f4c34105800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 0x7f4c34103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 0x7f4c34104380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3410d370 con 0x7f4c34103420 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3410d1f0 con 0x7f4c34102a60 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c39893640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3410d4f0 con 0x7f4c34101870 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 0x7f4c34104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 0x7f4c34104380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36600/0 (socket says 172.21.15.1:36600) 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.531+0000 7f4c337fe640 1 -- 172.21.15.1:0/3529377264 learned_addr learned my addr 172.21.15.1:0/3529377264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c32ffd640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34101870 0x7f4c34105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c2bfff640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 0x7f4c34103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c337fe640 1 -- 172.21.15.1:0/3529377264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34101870 msgr2=0x7f4c34105800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c337fe640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34101870 0x7f4c34105800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c337fe640 1 -- 172.21.15.1:0/3529377264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 msgr2=0x7f4c34103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c337fe640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 0x7f4c34103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c337fe640 1 -- 172.21.15.1:0/3529377264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c34104c10 con 0x7f4c34103420 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c2bfff640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 0x7f4c34103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c32ffd640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34101870 0x7f4c34105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c337fe640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 0x7f4c34104380 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00cad0 tx=0x7f4c2c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00ae20 con 0x7f4c34103420 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c2c005020 con 0x7f4c34103420 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c341a2d50 con 0x7f4c34103420 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00f6b0 con 0x7f4c34103420 2024-04-26T22:47:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.532+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c341a3190 con 0x7f4c34103420 2024-04-26T22:47:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.533+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c341a3480 con 0x7f4c34103420 2024-04-26T22:47:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.534+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4c2c0051d0 con 0x7f4c34103420 2024-04-26T22:47:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.535+0000 7f4c38891640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4c200a3bb0 0x7f4c200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.535+0000 7f4c32ffd640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4c200a3bb0 0x7f4c200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.535+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f4c2c0d04c0 con 0x7f4c34103420 2024-04-26T22:47:42.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.535+0000 7f4c32ffd640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4c200a3bb0 0x7f4c200a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c24006010 tx=0x7f4c24000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:42.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.539+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c2c014030 con 0x7f4c34103420 2024-04-26T22:47:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.767+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4c341a3750 con 0x7f4c34103420 2024-04-26T22:47:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.791+0000 7f4c38891640 1 -- 172.21.15.1:0/3529377264 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f4c2c09cf50 con 0x7f4c34103420 2024-04-26T22:47:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4c200a3bb0 msgr2=0x7f4c200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4c200a3bb0 0x7f4c200a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c24006010 tx=0x7f4c24000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 msgr2=0x7f4c34104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 0x7f4c34104380 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00cad0 tx=0x7f4c2c00cf90 comp rx=0 tx=0).stop 2024-04-26T22:47:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 shutdown_connections 2024-04-26T22:47:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4c200a3bb0 0x7f4c200a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c34103420 0x7f4c34104380 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c34102a60 0x7f4c34103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 --2- 172.21.15.1:0/3529377264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c34101870 0x7f4c34105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 >> 172.21.15.1:0/3529377264 conn(0x7f4c340fce00 msgr2=0x7f4c34069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 shutdown_connections 2024-04-26T22:47:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.794+0000 7f4c39893640 1 -- 172.21.15.1:0/3529377264 wait complete. 2024-04-26T22:47:42.805 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-26T22:47:42.805 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-26T22:47:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 -- 172.21.15.1:0/2149202006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0108650 msgr2=0x7f71a010aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 --2- 172.21.15.1:0/2149202006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0108650 0x7f71a010aaa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7190009ca0 tx=0x7f719002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 -- 172.21.15.1:0/2149202006 shutdown_connections 2024-04-26T22:47:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 --2- 172.21.15.1:0/2149202006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0105d20 0x7f71a0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 --2- 172.21.15.1:0/2149202006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0108650 0x7f71a010aaa0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 --2- 172.21.15.1:0/2149202006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a01033f0 0x7f71a01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 -- 172.21.15.1:0/2149202006 >> 172.21.15.1:0/2149202006 conn(0x7f71a00fce00 msgr2=0x7f71a00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.909+0000 7f71a5367640 1 -- 172.21.15.1:0/2149202006 shutdown_connections 2024-04-26T22:47:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 -- 172.21.15.1:0/2149202006 wait complete. 2024-04-26T22:47:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 Processor -- start 2024-04-26T22:47:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 -- start start 2024-04-26T22:47:42.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a01033f0 0x7f71a0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 0x7f71a0199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0108650 0x7f71a01a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f71a010e4c0 con 0x7f71a0105d20 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f71a010e340 con 0x7f71a01033f0 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.910+0000 7f71a5367640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71a010e640 con 0x7f71a0108650 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 0x7f71a0199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 0x7f71a0199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36628/0 (socket says 172.21.15.1:36628) 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 -- 172.21.15.1:0/3639192557 learned_addr learned my addr 172.21.15.1:0/3639192557 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719f7fe640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0108650 0x7f71a01a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719effd640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a01033f0 0x7f71a0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 -- 172.21.15.1:0/3639192557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0108650 msgr2=0x7f71a01a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0108650 0x7f71a01a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 -- 172.21.15.1:0/3639192557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a01033f0 msgr2=0x7f71a0198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a01033f0 0x7f71a0198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 -- 172.21.15.1:0/3639192557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a01a07e0 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719f7fe640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0108650 0x7f71a01a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f719e7fc640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 0x7f71a0199100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f718800c970 tx=0x7f718800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188007bb0 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a01a0ad0 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7188007d50 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188018730 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.911+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a01a0fd0 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.912+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a00fe8f0 con 0x7f71a0105d20 2024-04-26T22:47:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.913+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f718800f990 con 0x7f71a0105d20 2024-04-26T22:47:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.914+0000 7f7197fff640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f718c0a3bb0 0x7f718c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.914+0000 7f719effd640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f718c0a3bb0 0x7f718c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.914+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f71880d03a0 con 0x7f71a0105d20 2024-04-26T22:47:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.914+0000 7f719effd640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f718c0a3bb0 0x7f718c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7198005fd0 tx=0x7f7198005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:42.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:42.918+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7188014030 con 0x7f71a0105d20 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: pgmap v289: 97 pgs: 27 creating+peering, 8 active+clean, 62 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/376441185' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3199307278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:43.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2531752841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: pgmap v289: 97 pgs: 27 creating+peering, 8 active+clean, 62 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T22:47:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T22:47:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/376441185' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T22:47:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T22:47:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T22:47:43.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3199307278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:43.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2531752841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.146+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f71a0063530 con 0x7f71a0105d20 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: pgmap v289: 97 pgs: 27 creating+peering, 8 active+clean, 62 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/376441185' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3199307278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2531752841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:43.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.799+0000 7f7197fff640 1 -- 172.21.15.1:0/3639192557 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f718809c7e0 con 0x7f71a0105d20 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.801+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f718c0a3bb0 msgr2=0x7f718c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.801+0000 7f71a5367640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f718c0a3bb0 0x7f718c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7198005fd0 tx=0x7f7198005c70 comp rx=0 tx=0).stop 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 msgr2=0x7f71a0199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 0x7f71a0199100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f718800c970 tx=0x7f718800ce30 comp rx=0 tx=0).stop 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 shutdown_connections 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f718c0a3bb0 0x7f718c0a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a0108650 0x7f71a01a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71a0105d20 0x7f71a0199100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 --2- 172.21.15.1:0/3639192557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a01033f0 0x7f71a0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 >> 172.21.15.1:0/3639192557 conn(0x7f71a00fce00 msgr2=0x7f71a0103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 shutdown_connections 2024-04-26T22:47:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.802+0000 7f71a5367640 1 -- 172.21.15.1:0/3639192557 wait complete. 2024-04-26T22:47:43.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 -- 172.21.15.1:0/654299775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 msgr2=0x7f1350108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 --2- 172.21.15.1:0/654299775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f1350108120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1344009900 tx=0x7f134402a410 comp rx=0 tx=0).stop 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 -- 172.21.15.1:0/654299775 shutdown_connections 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 --2- 172.21.15.1:0/654299775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1350108660 0x7f135010aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 --2- 172.21.15.1:0/654299775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f1350108120 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 --2- 172.21.15.1:0/654299775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350103400 0x7f13501057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 -- 172.21.15.1:0/654299775 >> 172.21.15.1:0/654299775 conn(0x7f13500fce10 msgr2=0x7f13500ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 -- 172.21.15.1:0/654299775 shutdown_connections 2024-04-26T22:47:43.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.929+0000 7f1354ff7640 1 -- 172.21.15.1:0/654299775 wait complete. 2024-04-26T22:47:43.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 Processor -- start 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 -- start start 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1350103400 0x7f1350198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f13501990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350108660 0x7f13501a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f135010e4a0 con 0x7f1350105d30 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f135010e320 con 0x7f1350108660 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f1354ff7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f135010e620 con 0x7f1350103400 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.930+0000 7f134dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f13501990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f13501990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36658/0 (socket says 172.21.15.1:36658) 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 -- 172.21.15.1:0/1599788411 learned_addr learned my addr 172.21.15.1:0/1599788411 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134ed9d640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350108660 0x7f13501a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134e59c640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1350103400 0x7f1350198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 -- 172.21.15.1:0/1599788411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1350103400 msgr2=0x7f1350198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1350103400 0x7f1350198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 -- 172.21.15.1:0/1599788411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350108660 msgr2=0x7f13501a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350108660 0x7f13501a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 -- 172.21.15.1:0/1599788411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13501a0780 con 0x7f1350105d30 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f134dd9b640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f13501990a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1344006010 tx=0x7f13440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134403f030 con 0x7f1350105d30 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1344003cb0 con 0x7f1350105d30 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13501a0a10 con 0x7f1350105d30 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344035070 con 0x7f1350105d30 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.931+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13501a0f50 con 0x7f1350105d30 2024-04-26T22:47:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.932+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13500fe900 con 0x7f1350105d30 2024-04-26T22:47:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.933+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1344042030 con 0x7f1350105d30 2024-04-26T22:47:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.934+0000 7f133f7fe640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13380a3c30 0x7f13380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.934+0000 7f134e59c640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13380a3c30 0x7f13380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.934+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f13440f0850 con 0x7f1350105d30 2024-04-26T22:47:43.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.935+0000 7f134e59c640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13380a3c30 0x7f13380a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1340005fd0 tx=0x7f1340005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:43.939+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f13440bd380 con 0x7f1350105d30 2024-04-26T22:47:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:43 smithi001 ceph-mon[29435]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T22:47:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:43 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3529377264' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T22:47:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:43 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3529377264' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T22:47:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:43 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] up:active 2024-04-26T22:47:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:43 smithi001 ceph-mon[29435]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T22:47:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:43 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3639192557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T22:47:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:43 smithi038 ceph-mon[32433]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T22:47:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:43 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3529377264' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T22:47:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:43 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3529377264' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T22:47:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:43 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] up:active 2024-04-26T22:47:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:43 smithi038 ceph-mon[32433]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T22:47:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:43 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3639192557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T22:47:44.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.165+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f135002dd20 con 0x7f1350105d30 2024-04-26T22:47:44.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:43 smithi186 ceph-mon[32774]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T22:47:44.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:43 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3529377264' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T22:47:44.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:43 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3529377264' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T22:47:44.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:43 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] up:active 2024-04-26T22:47:44.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:43 smithi186 ceph-mon[32774]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T22:47:44.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:43 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3639192557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T22:47:44.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.805+0000 7f133f7fe640 1 -- 172.21.15.1:0/1599788411 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f13440bcc80 con 0x7f1350105d30 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13380a3c30 msgr2=0x7f13380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13380a3c30 0x7f13380a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1340005fd0 tx=0x7f1340005850 comp rx=0 tx=0).stop 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 msgr2=0x7f13501990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f13501990a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1344006010 tx=0x7f13440042f0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 shutdown_connections 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13380a3c30 0x7f13380a60f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350108660 0x7f13501a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1350105d30 0x7f13501990a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 --2- 172.21.15.1:0/1599788411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1350103400 0x7f1350198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 >> 172.21.15.1:0/1599788411 conn(0x7f13500fce10 msgr2=0x7f1350103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 shutdown_connections 2024-04-26T22:47:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.808+0000 7f1354ff7640 1 -- 172.21.15.1:0/1599788411 wait complete. 2024-04-26T22:47:44.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 -- 172.21.15.1:0/1087706675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c101880 msgr2=0x7fba3c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 --2- 172.21.15.1:0/1087706675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c101880 0x7fba3c101ca0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fba38009900 tx=0x7fba3802a410 comp rx=0 tx=0).stop 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 -- 172.21.15.1:0/1087706675 shutdown_connections 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 --2- 172.21.15.1:0/1087706675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba3c103430 0x7fba3c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 --2- 172.21.15.1:0/1087706675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c102a70 0x7fba3c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 --2- 172.21.15.1:0/1087706675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c101880 0x7fba3c101ca0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 -- 172.21.15.1:0/1087706675 >> 172.21.15.1:0/1087706675 conn(0x7fba3c0fce10 msgr2=0x7fba3c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 -- 172.21.15.1:0/1087706675 shutdown_connections 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.937+0000 7fba438d1640 1 -- 172.21.15.1:0/1087706675 wait complete. 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 Processor -- start 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 -- start start 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba3c101880 0x7fba3c10c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 0x7fba3c10a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c103430 0x7fba3c10ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10e4d0 con 0x7fba3c102a70 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10e350 con 0x7fba3c103430 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba438d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10e650 con 0x7fba3c101880 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 0x7fba3c10a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 0x7fba3c10a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36680/0 (socket says 172.21.15.1:36680) 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 -- 172.21.15.1:0/2278501779 learned_addr learned my addr 172.21.15.1:0/2278501779 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba4166d640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba3c101880 0x7fba3c10c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba41e6e640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c103430 0x7fba3c10ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 -- 172.21.15.1:0/2278501779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba3c101880 msgr2=0x7fba3c10c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba3c101880 0x7fba3c10c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 -- 172.21.15.1:0/2278501779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c103430 msgr2=0x7fba3c10ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c103430 0x7fba3c10ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.938+0000 7fba40e6c640 1 -- 172.21.15.1:0/2278501779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c10b610 con 0x7fba3c102a70 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.939+0000 7fba40e6c640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 0x7fba3c10a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00d950 tx=0x7fba2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.939+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c017070 con 0x7fba3c102a70 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.939+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba2c009c30 con 0x7fba3c102a70 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.939+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c1a2cf0 con 0x7fba3c102a70 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.939+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00f620 con 0x7fba3c102a70 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.939+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c1a3190 con 0x7fba3c102a70 2024-04-26T22:47:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.940+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3c108c70 con 0x7fba3c102a70 2024-04-26T22:47:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.941+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fba2c005030 con 0x7fba3c102a70 2024-04-26T22:47:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.941+0000 7fba327fc640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fba280a3bb0 0x7fba280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.942+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fba2c0cfe10 con 0x7fba3c102a70 2024-04-26T22:47:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.942+0000 7fba4166d640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fba280a3bb0 0x7fba280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.942+0000 7fba4166d640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fba280a3bb0 0x7fba280a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba38009230 tx=0x7fba380314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:44.945+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fba2c009de0 con 0x7fba3c102a70 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: pgmap v292: 97 pgs: 27 creating+peering, 36 active+clean, 34 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: Cluster is now healthy 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3639192557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T22:47:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-04-26T22:47:45.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T22:47:45.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:44 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1599788411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T22:47:45.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: pgmap v292: 97 pgs: 27 creating+peering, 36 active+clean, 34 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-26T22:47:45.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T22:47:45.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: Cluster is now healthy 2024-04-26T22:47:45.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T22:47:45.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3639192557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-26T22:47:45.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T22:47:45.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T22:47:45.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-04-26T22:47:45.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T22:47:45.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:44 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1599788411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T22:47:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.170+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fba3c02dbd0 con 0x7fba3c102a70 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: pgmap v292: 97 pgs: 27 creating+peering, 36 active+clean, 34 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: Cluster is now healthy 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3639192557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T22:47:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:44 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1599788411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T22:47:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.816+0000 7fba327fc640 1 -- 172.21.15.1:0/2278501779 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fba2c09c2f0 con 0x7fba3c102a70 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fba280a3bb0 msgr2=0x7fba280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fba280a3bb0 0x7fba280a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba38009230 tx=0x7fba380314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 msgr2=0x7fba3c10a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 0x7fba3c10a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00d950 tx=0x7fba2c00de10 comp rx=0 tx=0).stop 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 shutdown_connections 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fba280a3bb0 0x7fba280a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c103430 0x7fba3c10ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3c102a70 0x7fba3c10a840 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 --2- 172.21.15.1:0/2278501779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba3c101880 0x7fba3c10c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 >> 172.21.15.1:0/2278501779 conn(0x7fba3c0fce10 msgr2=0x7fba3c105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 shutdown_connections 2024-04-26T22:47:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.819+0000 7fba438d1640 1 -- 172.21.15.1:0/2278501779 wait complete. 2024-04-26T22:47:45.835 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-26T22:47:45.835 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3633248433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4101880 msgr2=0x7fe2b4101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3633248433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4101880 0x7fe2b4101ca0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009900 tx=0x7fe2ac02a410 comp rx=0 tx=0).stop 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3633248433 shutdown_connections 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3633248433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 0x7fe2b4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3633248433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4102a70 0x7fe2b4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3633248433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4101880 0x7fe2b4101ca0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3633248433 >> 172.21.15.1:0/3633248433 conn(0x7fe2b40fce10 msgr2=0x7fe2b40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3633248433 shutdown_connections 2024-04-26T22:47:45.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.941+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3633248433 wait complete. 2024-04-26T22:47:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 Processor -- start 2024-04-26T22:47:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 -- start start 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4101880 0x7fe2b4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 0x7fe2b4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 0x7fe2b41a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b410e4d0 con 0x7fe2b4102a70 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b410e350 con 0x7fe2b4101880 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b8d89640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b410e650 con 0x7fe2b4103430 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 0x7fe2b4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 0x7fe2b41a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 0x7fe2b4199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36698/0 (socket says 172.21.15.1:36698) 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.942+0000 7fe2b1d9b640 1 -- 172.21.15.1:0/3215824045 learned_addr learned my addr 172.21.15.1:0/3215824045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b259c640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4101880 0x7fe2b4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b2d9d640 1 -- 172.21.15.1:0/3215824045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4101880 msgr2=0x7fe2b4198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b2d9d640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4101880 0x7fe2b4198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b2d9d640 1 -- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 msgr2=0x7fe2b4199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b2d9d640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 0x7fe2b4199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b2d9d640 1 -- 172.21.15.1:0/3215824045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b41a0800 con 0x7fe2b4103430 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b259c640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4101880 0x7fe2b4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b1d9b640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 0x7fe2b4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b2d9d640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 0x7fe2b41a0190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009fd0 tx=0x7fe2a400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400ce20 con 0x7fe2b4103430 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b41a0af0 con 0x7fe2b4103430 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2a4005020 con 0x7fe2b4103430 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4010690 con 0x7fe2b4103430 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.943+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b41a1090 con 0x7fe2b4103430 2024-04-26T22:47:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.944+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b4108c10 con 0x7fe2b4103430 2024-04-26T22:47:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.945+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe2a40054d0 con 0x7fe2b4103430 2024-04-26T22:47:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.946+0000 7fe2a37fe640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe29c0a3bb0 0x7fe29c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.946+0000 7fe2b259c640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe29c0a3bb0 0x7fe29c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.946+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe2a40d0e00 con 0x7fe2b4103430 2024-04-26T22:47:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.946+0000 7fe2b259c640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe29c0a3bb0 0x7fe29c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac000c00 tx=0x7fe2ac0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:45.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:45.949+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe2a409d930 con 0x7fe2b4103430 2024-04-26T22:47:46.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1599788411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T22:47:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] up:active 2024-04-26T22:47:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T22:47:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-04-26T22:47:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-04-26T22:47:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-26T22:47:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:45 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2278501779' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T22:47:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1599788411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T22:47:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] up:active 2024-04-26T22:47:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T22:47:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-04-26T22:47:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-04-26T22:47:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-26T22:47:46.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:45 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2278501779' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T22:47:46.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.165+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe2b4062d70 con 0x7fe29c0a3bb0 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1599788411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] up:active 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-26T22:47:46.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:45 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2278501779' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T22:47:46.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:47:46.170+0000 7fd3fc2d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T22:47:46.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:47:46.170+0000 7fd3fc2d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T22:47:46.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:47:46.170+0000 7fd3fc2d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T22:47:46.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:47:46.170+0000 7fd3fc2d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T22:47:46.596 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T22:47:46.170+0000 7fd3fc2d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T22:47:46.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.822+0000 7fe2a37fe640 1 -- 172.21.15.1:0/3215824045 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe2b4062d70 con 0x7fe29c0a3bb0 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe29c0a3bb0 msgr2=0x7fe29c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe29c0a3bb0 0x7fe29c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac000c00 tx=0x7fe2ac0314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 msgr2=0x7fe2b41a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 0x7fe2b41a0190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009fd0 tx=0x7fe2a400ef10 comp rx=0 tx=0).stop 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 shutdown_connections 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe29c0a3bb0 0x7fe29c0a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2b4103430 0x7fe2b41a0190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b4102a70 0x7fe2b4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 --2- 172.21.15.1:0/3215824045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2b4101880 0x7fe2b4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 >> 172.21.15.1:0/3215824045 conn(0x7fe2b40fce10 msgr2=0x7fe2b4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 shutdown_connections 2024-04-26T22:47:46.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.825+0000 7fe2b8d89640 1 -- 172.21.15.1:0/3215824045 wait complete. 2024-04-26T22:47:46.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 -- 172.21.15.1:0/1956278381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 msgr2=0x7f7b8410aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 --2- 172.21.15.1:0/1956278381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b8410aab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009ca0 tx=0x7f7b7c02a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 -- 172.21.15.1:0/1956278381 shutdown_connections 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 --2- 172.21.15.1:0/1956278381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b8410aab0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 --2- 172.21.15.1:0/1956278381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84105d90 0x7f7b84108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 --2- 172.21.15.1:0/1956278381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84103460 0x7f7b84105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 -- 172.21.15.1:0/1956278381 >> 172.21.15.1:0/1956278381 conn(0x7f7b840fce10 msgr2=0x7f7b840ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 -- 172.21.15.1:0/1956278381 shutdown_connections 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.951+0000 7f7b898fc640 1 -- 172.21.15.1:0/1956278381 wait complete. 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 Processor -- start 2024-04-26T22:47:46.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 -- start start 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84103460 0x7f7b84198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84105d90 0x7f7b841990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b841a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8410e4d0 con 0x7f7b841086c0 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8410e350 con 0x7f7b84103460 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b898fc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8410e650 con 0x7f7b84105d90 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b841a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.952+0000 7f7b837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b841a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36726/0 (socket says 172.21.15.1:36726) 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b7a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84105d90 0x7f7b841990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 -- 172.21.15.1:0/240232561 learned_addr learned my addr 172.21.15.1:0/240232561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b82ffd640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84103460 0x7f7b84198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 -- 172.21.15.1:0/240232561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84105d90 msgr2=0x7f7b841990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84105d90 0x7f7b841990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 -- 172.21.15.1:0/240232561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84103460 msgr2=0x7f7b84198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84103460 0x7f7b84198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 -- 172.21.15.1:0/240232561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b841a07c0 con 0x7f7b841086c0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b82ffd640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84103460 0x7f7b84198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b7a7fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84105d90 0x7f7b841990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b837fe640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b841a0150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0097b0 tx=0x7f7b7c031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c003b70 con 0x7f7b841086c0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b7c003d10 con 0x7f7b841086c0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b841a0a50 con 0x7f7b841086c0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c02fc10 con 0x7f7b841086c0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.953+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b841a0fa0 con 0x7f7b841086c0 2024-04-26T22:47:46.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.954+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b840fe900 con 0x7f7b841086c0 2024-04-26T22:47:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.955+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7b7c004b40 con 0x7f7b841086c0 2024-04-26T22:47:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.956+0000 7f7b888fa640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7b700a3bb0 0x7f7b700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.956+0000 7f7b82ffd640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7b700a3bb0 0x7f7b700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.956+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7b7c0f1930 con 0x7f7b841086c0 2024-04-26T22:47:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.957+0000 7f7b82ffd640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7b700a3bb0 0x7f7b700a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009fc0 tx=0x7f7b74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:46.960+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b7c0be460 con 0x7f7b841086c0 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: pgmap v293: 97 pgs: 16 creating+peering, 81 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s wr, 11 op/s 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2278501779' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] up:active 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=h=up:creating} 8 up:standby 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] up:active 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: Cluster is now healthy 2024-04-26T22:47:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-04-26T22:47:47.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-26T22:47:47.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:46 smithi001 ceph-mon[29435]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-26T22:47:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: pgmap v293: 97 pgs: 16 creating+peering, 81 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s wr, 11 op/s 2024-04-26T22:47:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2278501779' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T22:47:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] up:active 2024-04-26T22:47:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-26T22:47:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=h=up:creating} 8 up:standby 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] up:active 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: Cluster is now healthy 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-26T22:47:47.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:46 smithi038 ceph-mon[32433]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-26T22:47:47.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.177+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7b84062d70 con 0x7f7b700a3bb0 2024-04-26T22:47:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.200+0000 7f7b888fa640 1 -- 172.21.15.1:0/240232561 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7b84062d70 con 0x7f7b700a3bb0 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7b700a3bb0 msgr2=0x7f7b700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7b700a3bb0 0x7f7b700a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009fc0 tx=0x7f7b74005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 msgr2=0x7f7b841a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b841a0150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0097b0 tx=0x7f7b7c031770 comp rx=0 tx=0).stop 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 shutdown_connections 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f7b700a3bb0 0x7f7b700a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b84105d90 0x7f7b841990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b841086c0 0x7f7b841a0150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 --2- 172.21.15.1:0/240232561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b84103460 0x7f7b84198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.202+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 >> 172.21.15.1:0/240232561 conn(0x7f7b840fce10 msgr2=0x7f7b84103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.203+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 shutdown_connections 2024-04-26T22:47:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.203+0000 7f7b898fc640 1 -- 172.21.15.1:0/240232561 wait complete. 2024-04-26T22:47:47.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: pgmap v293: 97 pgs: 16 creating+peering, 81 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s wr, 11 op/s 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2278501779' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] up:active 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=h=up:creating} 8 up:standby 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] up:active 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: Cluster is now healthy 2024-04-26T22:47:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-04-26T22:47:47.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-26T22:47:47.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:46 smithi186 ceph-mon[32774]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 -- 172.21.15.1:0/2792249368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090101880 msgr2=0x7f4090101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 --2- 172.21.15.1:0/2792249368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090101880 0x7f4090101ca0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4084009900 tx=0x7f408402a410 comp rx=0 tx=0).stop 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 -- 172.21.15.1:0/2792249368 shutdown_connections 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 --2- 172.21.15.1:0/2792249368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4090102a70 0x7f4090102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 --2- 172.21.15.1:0/2792249368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4090103430 0x7f4090109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 --2- 172.21.15.1:0/2792249368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090101880 0x7f4090101ca0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 -- 172.21.15.1:0/2792249368 >> 172.21.15.1:0/2792249368 conn(0x7f40900fce10 msgr2=0x7f40900ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 -- 172.21.15.1:0/2792249368 shutdown_connections 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.320+0000 7f4097146640 1 -- 172.21.15.1:0/2792249368 wait complete. 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 Processor -- start 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 -- start start 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4090101880 0x7f409010c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 0x7f409010a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4090103430 0x7f409010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f409010e4d0 con 0x7f4090102a70 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f409010e350 con 0x7f4090101880 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4097146640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f409010e650 con 0x7f4090103430 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f408ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 0x7f409010a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f408ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 0x7f409010a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36742/0 (socket says 172.21.15.1:36742) 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f40956e3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4090103430 0x7f409010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f408ffff640 1 -- 172.21.15.1:0/2340117100 learned_addr learned my addr 172.21.15.1:0/2340117100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f4094ee2640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4090101880 0x7f409010c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f408ffff640 1 -- 172.21.15.1:0/2340117100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4090103430 msgr2=0x7f409010ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.321+0000 7f408ffff640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4090103430 0x7f409010ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408ffff640 1 -- 172.21.15.1:0/2340117100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4090101880 msgr2=0x7f409010c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408ffff640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4090101880 0x7f409010c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408ffff640 1 -- 172.21.15.1:0/2340117100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409010b610 con 0x7f4090102a70 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408ffff640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 0x7f409010a840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f407c00e9f0 tx=0x7f407c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c00cd80 con 0x7f4090102a70 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f407c0054d0 con 0x7f4090102a70 2024-04-26T22:47:47.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40901a2d10 con 0x7f4090102a70 2024-04-26T22:47:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c010690 con 0x7f4090102a70 2024-04-26T22:47:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.322+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40901a31b0 con 0x7f4090102a70 2024-04-26T22:47:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.323+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4090108c20 con 0x7f4090102a70 2024-04-26T22:47:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.324+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f407c004e90 con 0x7f4090102a70 2024-04-26T22:47:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.325+0000 7f408dffb640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40800a3c30 0x7f40800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.325+0000 7f4094ee2640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40800a3c30 0x7f40800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.325+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f407c0d07c0 con 0x7f4090102a70 2024-04-26T22:47:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.325+0000 7f4094ee2640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40800a3c30 0x7f40800a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4084000c00 tx=0x7f40840314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.329+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f407c09d250 con 0x7f4090102a70 2024-04-26T22:47:47.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.547+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f4090062e00 con 0x7f40800a3c30 2024-04-26T22:47:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.553+0000 7f408dffb640 1 -- 172.21.15.1:0/2340117100 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4090062e00 con 0x7f40800a3c30 2024-04-26T22:47:47.553 INFO:teuthology.orchestra.run.smithi001.stdout:/volumes/qa/sv_0/76542baf-88e9-4b30-9ca5-64cf54f69f59 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40800a3c30 msgr2=0x7f40800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40800a3c30 0x7f40800a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4084000c00 tx=0x7f40840314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 msgr2=0x7f409010a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 0x7f409010a840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f407c00e9f0 tx=0x7f407c00edf0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 shutdown_connections 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40800a3c30 0x7f40800a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4090103430 0x7f409010ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4090102a70 0x7f409010a840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 --2- 172.21.15.1:0/2340117100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4090101880 0x7f409010c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 >> 172.21.15.1:0/2340117100 conn(0x7f40900fce10 msgr2=0x7f4090105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 shutdown_connections 2024-04-26T22:47:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.555+0000 7f4097146640 1 -- 172.21.15.1:0/2340117100 wait complete. 2024-04-26T22:47:47.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 -- 172.21.15.1:0/1373122713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70103b70 msgr2=0x7fcb70103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 --2- 172.21.15.1:0/1373122713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70103b70 0x7fcb70103ff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009900 tx=0x7fcb6402a410 comp rx=0 tx=0).stop 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 -- 172.21.15.1:0/1373122713 shutdown_connections 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 --2- 172.21.15.1:0/1373122713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb70104530 0x7fcb7010adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 --2- 172.21.15.1:0/1373122713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70103b70 0x7fcb70103ff0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 --2- 172.21.15.1:0/1373122713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70102980 0x7fcb70102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 -- 172.21.15.1:0/1373122713 >> 172.21.15.1:0/1373122713 conn(0x7fcb700fdef0 msgr2=0x7fcb70100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 -- 172.21.15.1:0/1373122713 shutdown_connections 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.673+0000 7fcb75b14640 1 -- 172.21.15.1:0/1373122713 wait complete. 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 Processor -- start 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 -- start start 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 0x7fcb70198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70103b70 0x7fcb70199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb70104530 0x7fcb701a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7010e490 con 0x7fcb70102980 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7010e310 con 0x7fcb70103b70 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb75b14640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7010e610 con 0x7fcb70104530 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 0x7fcb70198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 0x7fcb70198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36764/0 (socket says 172.21.15.1:36764) 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb6f7fe640 1 -- 172.21.15.1:0/3376860682 learned_addr learned my addr 172.21.15.1:0/3376860682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.674+0000 7fcb6ffff640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb70104530 0x7fcb701a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6effd640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70103b70 0x7fcb70199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6f7fe640 1 -- 172.21.15.1:0/3376860682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb70104530 msgr2=0x7fcb701a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6f7fe640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb70104530 0x7fcb701a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6f7fe640 1 -- 172.21.15.1:0/3376860682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70103b70 msgr2=0x7fcb70199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6f7fe640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70103b70 0x7fcb70199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6f7fe640 1 -- 172.21.15.1:0/3376860682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb701a0810 con 0x7fcb70102980 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6effd640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70103b70 0x7fcb70199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:47:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6f7fe640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 0x7fcb70198bf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000c970 tx=0x7fcb6000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:47.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60004220 con 0x7fcb70102980 2024-04-26T22:47:47.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb600043c0 con 0x7fcb70102980 2024-04-26T22:47:47.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb701a0b00 con 0x7fcb70102980 2024-04-26T22:47:47.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60003e30 con 0x7fcb70102980 2024-04-26T22:47:47.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.675+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb701a1050 con 0x7fcb70102980 2024-04-26T22:47:47.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.676+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb70109e10 con 0x7fcb70102980 2024-04-26T22:47:47.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.677+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fcb600079b0 con 0x7fcb70102980 2024-04-26T22:47:47.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.678+0000 7fcb6cff9640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb580a3bb0 0x7fcb580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:47.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.678+0000 7fcb6effd640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb580a3bb0 0x7fcb580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:47.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.678+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcb600d0430 con 0x7fcb70102980 2024-04-26T22:47:47.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.678+0000 7fcb6effd640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb580a3bb0 0x7fcb580a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcb64006010 tx=0x7fcb64000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:47.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.682+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcb60014030 con 0x7fcb70102980 2024-04-26T22:47:47.827 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T22:47:47.556+0000 7f79095f7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T22:47:47.827 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T22:47:47.556+0000 7f79095f7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T22:47:47.827 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T22:47:47.556+0000 7f79095f7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T22:47:47.827 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T22:47:47.556+0000 7f79095f7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-26T22:47:47.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.903+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcb70062d70 con 0x7fcb580a3bb0 2024-04-26T22:47:47.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.927+0000 7fcb6cff9640 1 -- 172.21.15.1:0/3376860682 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcb70062d70 con 0x7fcb580a3bb0 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb580a3bb0 msgr2=0x7fcb580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb580a3bb0 0x7fcb580a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcb64006010 tx=0x7fcb64000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 msgr2=0x7fcb70198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 0x7fcb70198bf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000c970 tx=0x7fcb6000cd70 comp rx=0 tx=0).stop 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 shutdown_connections 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb580a3bb0 0x7fcb580a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb70104530 0x7fcb701a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb70103b70 0x7fcb70199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 --2- 172.21.15.1:0/3376860682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb70102980 0x7fcb70198bf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 >> 172.21.15.1:0/3376860682 conn(0x7fcb700fdef0 msgr2=0x7fcb70106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 shutdown_connections 2024-04-26T22:47:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:47.929+0000 7fcb75b14640 1 -- 172.21.15.1:0/3376860682 wait complete. 2024-04-26T22:47:47.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T22:47:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 -- 172.21.15.1:0/3429965356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c04105dc0 msgr2=0x7f5c041081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 --2- 172.21.15.1:0/3429965356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c04105dc0 0x7f5c041081b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009900 tx=0x7f5bf802a410 comp rx=0 tx=0).stop 2024-04-26T22:47:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 -- 172.21.15.1:0/3429965356 shutdown_connections 2024-04-26T22:47:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 --2- 172.21.15.1:0/3429965356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c041086f0 0x7f5c0410aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 --2- 172.21.15.1:0/3429965356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c04105dc0 0x7f5c041081b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 --2- 172.21.15.1:0/3429965356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c04103490 0x7f5c04105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 -- 172.21.15.1:0/3429965356 >> 172.21.15.1:0/3429965356 conn(0x7f5c040fce00 msgr2=0x7f5c040ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 -- 172.21.15.1:0/3429965356 shutdown_connections 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 -- 172.21.15.1:0/3429965356 wait complete. 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 Processor -- start 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.047+0000 7f5c0a778640 1 -- start start 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 0x7f5c04198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c04105dc0 0x7f5c041990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 0x7f5c041a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0410e4f0 con 0x7f5c041086f0 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0410e370 con 0x7f5c04105dc0 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0410e670 con 0x7f5c04103490 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c08d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 0x7f5c041a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 0x7f5c04198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 0x7f5c04198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46366/0 (socket says 172.21.15.1:46366) 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 -- 172.21.15.1:0/2919949275 learned_addr learned my addr 172.21.15.1:0/2919949275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c08d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 0x7f5c041a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36780/0 (socket says 172.21.15.1:36780) 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c037fe640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c04105dc0 0x7f5c041990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 -- 172.21.15.1:0/2919949275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c04105dc0 msgr2=0x7f5c041990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c04105dc0 0x7f5c041990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 -- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 msgr2=0x7f5c041a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 0x7f5c041a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 -- 172.21.15.1:0/2919949275 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c041a07b0 con 0x7f5c04103490 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c08d15640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 0x7f5c041a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c03fff640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 0x7f5c04198b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c920 tx=0x7f5bf400cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4007c70 con 0x7f5c04103490 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c041a0aa0 con 0x7f5c04103490 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf4007e10 con 0x7f5c04103490 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.048+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4018820 con 0x7f5c04103490 2024-04-26T22:47:48.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.049+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c041a1010 con 0x7f5c04103490 2024-04-26T22:47:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.050+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c040fe8f0 con 0x7f5c04103490 2024-04-26T22:47:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.051+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5bf400f450 con 0x7f5c04103490 2024-04-26T22:47:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.051+0000 7f5c017fa640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5bec0a3bb0 0x7f5bec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.051+0000 7f5c037fe640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5bec0a3bb0 0x7f5bec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.051+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5bf40d03d0 con 0x7f5c04103490 2024-04-26T22:47:48.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.052+0000 7f5c037fe640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5bec0a3bb0 0x7f5bec0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8006010 tx=0x7f5bf80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:48.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.056+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5bf4014030 con 0x7f5c04103490 2024-04-26T22:47:48.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:47 smithi001 ceph-mon[29435]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:47 smithi001 ceph-mon[29435]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:47 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] up:active 2024-04-26T22:47:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:47 smithi001 ceph-mon[29435]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-26T22:47:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:47 smithi001 ceph-mon[29435]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T22:47:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:47 smithi038 ceph-mon[32433]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:47 smithi038 ceph-mon[32433]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:47 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] up:active 2024-04-26T22:47:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:47 smithi038 ceph-mon[32433]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-26T22:47:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:47 smithi038 ceph-mon[32433]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T22:47:48.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-mon[32774]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-mon[32774]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] up:active 2024-04-26T22:47:48.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-mon[32774]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-26T22:47:48.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:47 smithi186 ceph-mon[32774]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T22:47:48.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.273+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f5c04062d70 con 0x7f5bec0a3bb0 2024-04-26T22:47:48.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.277+0000 7f5c017fa640 1 -- 172.21.15.1:0/2919949275 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5c04062d70 con 0x7f5bec0a3bb0 2024-04-26T22:47:48.278 INFO:teuthology.orchestra.run.smithi001.stdout:/volumes/qa/sv_1/69154049-25b2-4b6a-a5ed-0e285f8f1614 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5bec0a3bb0 msgr2=0x7f5bec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5bec0a3bb0 0x7f5bec0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8006010 tx=0x7f5bf80314a0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 msgr2=0x7f5c04198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 0x7f5c04198b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c920 tx=0x7f5bf400cde0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 shutdown_connections 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5bec0a3bb0 0x7f5bec0a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c041086f0 0x7f5c041a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c04105dc0 0x7f5c041990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 --2- 172.21.15.1:0/2919949275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c04103490 0x7f5c04198b90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 >> 172.21.15.1:0/2919949275 conn(0x7f5c040fce00 msgr2=0x7f5c04103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.279+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 shutdown_connections 2024-04-26T22:47:48.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.280+0000 7f5c0a778640 1 -- 172.21.15.1:0/2919949275 wait complete. 2024-04-26T22:47:48.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:47:48.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 -- 172.21.15.1:0/50101436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 msgr2=0x7faf5c102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 --2- 172.21.15.1:0/50101436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c102020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faf50009900 tx=0x7faf5002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 -- 172.21.15.1:0/50101436 shutdown_connections 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 --2- 172.21.15.1:0/50101436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf5c102560 0x7faf5c1029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 --2- 172.21.15.1:0/50101436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c102020 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 --2- 172.21.15.1:0/50101436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c101260 0x7faf5c101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 -- 172.21.15.1:0/50101436 >> 172.21.15.1:0/50101436 conn(0x7faf5c0fce00 msgr2=0x7faf5c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 -- 172.21.15.1:0/50101436 shutdown_connections 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.392+0000 7faf61c72640 1 -- 172.21.15.1:0/50101436 wait complete. 2024-04-26T22:47:48.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 Processor -- start 2024-04-26T22:47:48.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 -- start start 2024-04-26T22:47:48.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf5c101260 0x7faf5c19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c102560 0x7faf5c1a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c1106a0 con 0x7faf5c101bc0 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c110520 con 0x7faf5c101260 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf61c72640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c110820 con 0x7faf5c102560 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c19b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36786/0 (socket says 172.21.15.1:36786) 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 -- 172.21.15.1:0/2009203996 learned_addr learned my addr 172.21.15.1:0/2009203996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5b7fe640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf5c101260 0x7faf5c19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5bfff640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c102560 0x7faf5c1a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 -- 172.21.15.1:0/2009203996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c102560 msgr2=0x7faf5c1a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c102560 0x7faf5c1a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 -- 172.21.15.1:0/2009203996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf5c101260 msgr2=0x7faf5c19ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf5c101260 0x7faf5c19ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.393+0000 7faf5affd640 1 -- 172.21.15.1:0/2009203996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c1a2990 con 0x7faf5c101bc0 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.394+0000 7faf5affd640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c19b2b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faf50005940 tx=0x7faf50031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.394+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5003f030 con 0x7faf5c101bc0 2024-04-26T22:47:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.394+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf50003aa0 con 0x7faf5c101bc0 2024-04-26T22:47:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.394+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5c1a2c20 con 0x7faf5c101bc0 2024-04-26T22:47:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.394+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50035070 con 0x7faf5c101bc0 2024-04-26T22:47:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.394+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5c1a3160 con 0x7faf5c101bc0 2024-04-26T22:47:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.395+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5c10c1a0 con 0x7faf5c101bc0 2024-04-26T22:47:48.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.396+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7faf50042030 con 0x7faf5c101bc0 2024-04-26T22:47:48.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.397+0000 7faf58ff9640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7faf440a3c30 0x7faf440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.397+0000 7faf5b7fe640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7faf440a3c30 0x7faf440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.397+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7faf500f0d80 con 0x7faf5c101bc0 2024-04-26T22:47:48.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.397+0000 7faf5b7fe640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7faf440a3c30 0x7faf440a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009bd0 tx=0x7faf4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:48.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.401+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faf500bd260 con 0x7faf5c101bc0 2024-04-26T22:47:48.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.628+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf5c02dd20 con 0x7faf5c101bc0 2024-04-26T22:47:48.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.628+0000 7faf58ff9640 1 -- 172.21.15.1:0/2009203996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11917 (secure 0 0 0) 0x7faf500c1560 con 0x7faf5c101bc0 2024-04-26T22:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:47:47.801060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[24475,24514,24487,24473,14631]},"id":1}]} 2024-04-26T22:47:48.629 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.630+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7faf440a3c30 msgr2=0x7faf440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.630+0000 7faf61c72640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7faf440a3c30 0x7faf440a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009bd0 tx=0x7faf4c005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.630+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 msgr2=0x7faf5c19b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.630+0000 7faf61c72640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c19b2b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faf50005940 tx=0x7faf50031a50 comp rx=0 tx=0).stop 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.630+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 shutdown_connections 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7faf440a3c30 0x7faf440a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c102560 0x7faf5c1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5c101bc0 0x7faf5c19b2b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 --2- 172.21.15.1:0/2009203996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf5c101260 0x7faf5c19ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 >> 172.21.15.1:0/2009203996 conn(0x7faf5c0fce00 msgr2=0x7faf5c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 shutdown_connections 2024-04-26T22:47:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.631+0000 7faf61c72640 1 -- 172.21.15.1:0/2009203996 wait complete. 2024-04-26T22:47:48.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:47:48.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/2586997891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 msgr2=0x7fc0c0105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/2586997891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0105800 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009900 tx=0x7fc0b002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/2586997891 shutdown_connections 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/2586997891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c0108670 0x7fc0c010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/2586997891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c0105d40 0x7fc0c0108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/2586997891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0105800 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/2586997891 >> 172.21.15.1:0/2586997891 conn(0x7fc0c00fce00 msgr2=0x7fc0c00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/2586997891 shutdown_connections 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.743+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/2586997891 wait complete. 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 Processor -- start 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 -- start start 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c0105d40 0x7fc0c0199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c0108670 0x7fc0c01a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c010e4c0 con 0x7fc0c0103410 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c010e340 con 0x7fc0c0105d40 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0c5ac4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c010e640 con 0x7fc0c0108670 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36802/0 (socket says 172.21.15.1:36802) 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.744+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/3409494320 learned_addr learned my addr 172.21.15.1:0/3409494320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0beffd640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c0105d40 0x7fc0c0199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bffff640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c0108670 0x7fc0c01a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/3409494320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c0108670 msgr2=0x7fc0c01a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c0108670 0x7fc0c01a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/3409494320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c0105d40 msgr2=0x7fc0c0199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c0105d40 0x7fc0c0199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/3409494320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c01a0810 con 0x7fc0c0103410 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0198bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0005940 tx=0x7fc0b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b003f030 con 0x7fc0c0103410 2024-04-26T22:47:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0b0003cb0 con 0x7fc0c0103410 2024-04-26T22:47:48.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c01a0aa0 con 0x7fc0c0103410 2024-04-26T22:47:48.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0035070 con 0x7fc0c0103410 2024-04-26T22:47:48.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.745+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c01a0f90 con 0x7fc0c0103410 2024-04-26T22:47:48.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.746+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c00fe8f0 con 0x7fc0c0103410 2024-04-26T22:47:48.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.747+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc0b0042030 con 0x7fc0c0103410 2024-04-26T22:47:48.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.748+0000 7fc0bcff9640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a80a3bb0 0x7fc0a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:48.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.748+0000 7fc0beffd640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a80a3bb0 0x7fc0a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:48.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.748+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc0b00f0710 con 0x7fc0c0103410 2024-04-26T22:47:48.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.748+0000 7fc0beffd640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a80a3bb0 0x7fc0a80a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0c019a100 tx=0x7fc0b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:48.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.753+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0b00bd1a0 con 0x7fc0c0103410 2024-04-26T22:47:48.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.818+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc0b00b9110 con 0x7fc0c0103410 2024-04-26T22:47:48.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.952+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0c0063530 con 0x7fc0c0103410 2024-04-26T22:47:48.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.954+0000 7fc0bcff9640 1 -- 172.21.15.1:0/3409494320 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22040 (secure 0 0 0) 0x7fc0b0003e60 con 0x7fc0c0103410 2024-04-26T22:47:48.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:48.954 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":85,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:47:42.763750+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a80a3bb0 msgr2=0x7fc0a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a80a3bb0 0x7fc0a80a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0c019a100 tx=0x7fc0b4005f90 comp rx=0 tx=0).stop 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 msgr2=0x7fc0c0198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0198bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0005940 tx=0x7fc0b00042f0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 shutdown_connections 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a80a3bb0 0x7fc0a80a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c0108670 0x7fc0c01a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c0105d40 0x7fc0c0199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 --2- 172.21.15.1:0/3409494320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c0103410 0x7fc0c0198bf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 >> 172.21.15.1:0/3409494320 conn(0x7fc0c00fce00 msgr2=0x7fc0c0103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 shutdown_connections 2024-04-26T22:47:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:48.956+0000 7fc0c5ac4640 1 -- 172.21.15.1:0/3409494320 wait complete. 2024-04-26T22:47:48.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:48 smithi001 ceph-mon[29435]: pgmap v294: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1 KiB/s rd, 5.7 KiB/s wr, 23 op/s 2024-04-26T22:47:48.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:48 smithi001 ceph-mon[29435]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:48 smithi001 ceph-mon[29435]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:48.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:48 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2009203996' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:48.966 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:48 smithi001 ceph-mon[29435]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T22:47:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T22:47:48.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T22:47:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.069+0000 7f3ee1732640 1 -- 172.21.15.1:0/1080857929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc102a80 msgr2=0x7f3edc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.069+0000 7f3ee1732640 1 --2- 172.21.15.1:0/1080857929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc102a80 0x7f3edc102f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009900 tx=0x7f3ed002a410 comp rx=0 tx=0).stop 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 -- 172.21.15.1:0/1080857929 shutdown_connections 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 --2- 172.21.15.1:0/1080857929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3edc103440 0x7f3edc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 --2- 172.21.15.1:0/1080857929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc102a80 0x7f3edc102f00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 --2- 172.21.15.1:0/1080857929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3edc101890 0x7f3edc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 -- 172.21.15.1:0/1080857929 >> 172.21.15.1:0/1080857929 conn(0x7f3edc0fce00 msgr2=0x7f3edc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 -- 172.21.15.1:0/1080857929 shutdown_connections 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 -- 172.21.15.1:0/1080857929 wait complete. 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 Processor -- start 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 -- start start 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 0x7f3edc198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3edc102a80 0x7f3edc1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3edc103440 0x7f3edc1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.070+0000 7f3ee1732640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc10e4e0 con 0x7f3edc101890 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ee1732640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc10e360 con 0x7f3edc103440 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ee1732640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc10e660 con 0x7f3edc102a80 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 0x7f3edc198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 0x7f3edc198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36826/0 (socket says 172.21.15.1:36826) 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 -- 172.21.15.1:0/2958035908 learned_addr learned my addr 172.21.15.1:0/2958035908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edb7fe640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3edc103440 0x7f3edc1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3eda7fc640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3edc102a80 0x7f3edc1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 -- 172.21.15.1:0/2958035908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3edc102a80 msgr2=0x7f3edc1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3edc102a80 0x7f3edc1990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 -- 172.21.15.1:0/2958035908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3edc103440 msgr2=0x7f3edc1a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3edc103440 0x7f3edc1a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 -- 172.21.15.1:0/2958035908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc1a0790 con 0x7f3edc101890 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3edaffd640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 0x7f3edc198b70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00d770 tx=0x7f3ecc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc017070 con 0x7f3edc101890 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ecc009c30 con 0x7f3edc101890 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc003cc0 con 0x7f3edc101890 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc1a0a80 con 0x7f3edc101890 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.071+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1a0fd0 con 0x7f3edc101890 2024-04-26T22:47:49.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.072+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc108c30 con 0x7f3edc101890 2024-04-26T22:47:49.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.073+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3ecc003e60 con 0x7f3edc101890 2024-04-26T22:47:49.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.074+0000 7f3ec3fff640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ec40a3c40 0x7f3ec40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.074+0000 7f3eda7fc640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ec40a3c40 0x7f3ec40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.074+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3ecc0cff20 con 0x7f3edc101890 2024-04-26T22:47:49.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.075+0000 7f3eda7fc640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ec40a3c40 0x7f3ec40a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3ed002a930 tx=0x7f3ed0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:49.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.079+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3ecc09c850 con 0x7f3edc101890 2024-04-26T22:47:49.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:48 smithi038 ceph-mon[32433]: pgmap v294: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1 KiB/s rd, 5.7 KiB/s wr, 23 op/s 2024-04-26T22:47:49.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:48 smithi038 ceph-mon[32433]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:49.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:48 smithi038 ceph-mon[32433]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:49.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:48 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2009203996' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:49.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:48 smithi038 ceph-mon[32433]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T22:47:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:48 smithi186 ceph-mon[32774]: pgmap v294: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1 KiB/s rd, 5.7 KiB/s wr, 23 op/s 2024-04-26T22:47:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:48 smithi186 ceph-mon[32774]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:48 smithi186 ceph-mon[32774]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:47:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:48 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2009203996' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:47:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:48 smithi186 ceph-mon[32774]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T22:47:49.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.263+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3edc062d70 con 0x7f3ec40a3c40 2024-04-26T22:47:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.271+0000 7f3ec3fff640 1 -- 172.21.15.1:0/2958035908 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258935 (secure 0 0 0) 0x7f3edc062d70 con 0x7f3ec40a3c40 2024-04-26T22:47:49.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ec40a3c40 msgr2=0x7f3ec40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ec40a3c40 0x7f3ec40a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3ed002a930 tx=0x7f3ed0000bc0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 msgr2=0x7f3edc198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 0x7f3edc198b70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00d770 tx=0x7f3ecc00dc30 comp rx=0 tx=0).stop 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 shutdown_connections 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ec40a3c40 0x7f3ec40a6100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3edc102a80 0x7f3edc1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3edc103440 0x7f3edc1a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 --2- 172.21.15.1:0/2958035908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3edc101890 0x7f3edc198b70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 >> 172.21.15.1:0/2958035908 conn(0x7f3edc0fce00 msgr2=0x7f3edc105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 shutdown_connections 2024-04-26T22:47:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.274+0000 7f3ee1732640 1 -- 172.21.15.1:0/2958035908 wait complete. 2024-04-26T22:47:49.286 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-26T22:47:47.751007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":133,"num_read_kb":116,"num_write":335,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":269,"ondisk_log_size":269,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":249,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360244,"kb_used_data":11060,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581580,"statfs":{"total":1151940427776,"available":1151571537920,"internally_reserved":0,"allocated":11325440,"data_stored":7040115,"data_compressed":109848,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5950,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.997722"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.780073+0000","last_change":"2024-04-26T22:47:41.777587+0000","last_active":"2024-04-26T22:47:41.780073+0000","last_peered":"2024-04-26T22:47:41.780073+0000","last_clean":"2024-04-26T22:47:41.780073+0000","last_became_active":"2024-04-26T22:47:41.777165+0000","last_became_peered":"2024-04-26T22:47:41.777165+0000","last_unstale":"2024-04-26T22:47:41.780073+0000","last_undegraded":"2024-04-26T22:47:41.780073+0000","last_fullsized":"2024-04-26T22:47:41.780073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:31.561664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451905+0000","last_change":"2024-04-26T22:47:41.776888+0000","last_active":"2024-04-26T22:47:43.451905+0000","last_peered":"2024-04-26T22:47:43.451905+0000","last_clean":"2024-04-26T22:47:43.451905+0000","last_became_active":"2024-04-26T22:47:41.776580+0000","last_became_peered":"2024-04-26T22:47:41.776580+0000","last_unstale":"2024-04-26T22:47:43.451905+0000","last_undegraded":"2024-04-26T22:47:43.451905+0000","last_fullsized":"2024-04-26T22:47:43.451905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:49.263390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451484+0000","last_change":"2024-04-26T22:47:41.784374+0000","last_active":"2024-04-26T22:47:43.451484+0000","last_peered":"2024-04-26T22:47:43.451484+0000","last_clean":"2024-04-26T22:47:43.451484+0000","last_became_active":"2024-04-26T22:47:41.783119+0000","last_became_peered":"2024-04-26T22:47:41.783119+0000","last_unstale":"2024-04-26T22:47:43.451484+0000","last_undegraded":"2024-04-26T22:47:43.451484+0000","last_fullsized":"2024-04-26T22:47:43.451484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:18.599515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452665+0000","last_change":"2024-04-26T22:47:41.782485+0000","last_active":"2024-04-26T22:47:43.452665+0000","last_peered":"2024-04-26T22:47:43.452665+0000","last_clean":"2024-04-26T22:47:43.452665+0000","last_became_active":"2024-04-26T22:47:41.782118+0000","last_became_peered":"2024-04-26T22:47:41.782118+0000","last_unstale":"2024-04-26T22:47:43.452665+0000","last_undegraded":"2024-04-26T22:47:43.452665+0000","last_fullsized":"2024-04-26T22:47:43.452665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:39.953297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451707+0000","last_change":"2024-04-26T22:47:41.776281+0000","last_active":"2024-04-26T22:47:43.451707+0000","last_peered":"2024-04-26T22:47:43.451707+0000","last_clean":"2024-04-26T22:47:43.451707+0000","last_became_active":"2024-04-26T22:47:41.775838+0000","last_became_peered":"2024-04-26T22:47:41.775838+0000","last_unstale":"2024-04-26T22:47:43.451707+0000","last_undegraded":"2024-04-26T22:47:43.451707+0000","last_fullsized":"2024-04-26T22:47:43.451707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:04.702698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767918+0000","last_change":"2024-04-26T22:47:41.785568+0000","last_active":"2024-04-26T22:47:42.767918+0000","last_peered":"2024-04-26T22:47:42.767918+0000","last_clean":"2024-04-26T22:47:42.767918+0000","last_became_active":"2024-04-26T22:47:41.785188+0000","last_became_peered":"2024-04-26T22:47:41.785188+0000","last_unstale":"2024-04-26T22:47:42.767918+0000","last_undegraded":"2024-04-26T22:47:42.767918+0000","last_fullsized":"2024-04-26T22:47:42.767918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:21.928641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180128+0000","last_change":"2024-04-26T22:47:41.785722+0000","last_active":"2024-04-26T22:47:43.180128+0000","last_peered":"2024-04-26T22:47:43.180128+0000","last_clean":"2024-04-26T22:47:43.180128+0000","last_became_active":"2024-04-26T22:47:41.784935+0000","last_became_peered":"2024-04-26T22:47:41.784935+0000","last_unstale":"2024-04-26T22:47:43.180128+0000","last_undegraded":"2024-04-26T22:47:43.180128+0000","last_fullsized":"2024-04-26T22:47:43.180128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:24.047598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451601+0000","last_change":"2024-04-26T22:47:41.777075+0000","last_active":"2024-04-26T22:47:43.451601+0000","last_peered":"2024-04-26T22:47:43.451601+0000","last_clean":"2024-04-26T22:47:43.451601+0000","last_became_active":"2024-04-26T22:47:41.776734+0000","last_became_peered":"2024-04-26T22:47:41.776734+0000","last_unstale":"2024-04-26T22:47:43.451601+0000","last_undegraded":"2024-04-26T22:47:43.451601+0000","last_fullsized":"2024-04-26T22:47:43.451601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:14.343905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451922+0000","last_change":"2024-04-26T22:47:41.775207+0000","last_active":"2024-04-26T22:47:43.451922+0000","last_peered":"2024-04-26T22:47:43.451922+0000","last_clean":"2024-04-26T22:47:43.451922+0000","last_became_active":"2024-04-26T22:47:41.774834+0000","last_became_peered":"2024-04-26T22:47:41.774834+0000","last_unstale":"2024-04-26T22:47:43.451922+0000","last_undegraded":"2024-04-26T22:47:43.451922+0000","last_fullsized":"2024-04-26T22:47:43.451922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:25.913385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.768518+0000","last_change":"2024-04-26T22:47:41.785894+0000","last_active":"2024-04-26T22:47:42.768518+0000","last_peered":"2024-04-26T22:47:42.768518+0000","last_clean":"2024-04-26T22:47:42.768518+0000","last_became_active":"2024-04-26T22:47:41.785213+0000","last_became_peered":"2024-04-26T22:47:41.785213+0000","last_unstale":"2024-04-26T22:47:42.768518+0000","last_undegraded":"2024-04-26T22:47:42.768518+0000","last_fullsized":"2024-04-26T22:47:42.768518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:44.927776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180604+0000","last_change":"2024-04-26T22:47:41.781532+0000","last_active":"2024-04-26T22:47:43.180604+0000","last_peered":"2024-04-26T22:47:43.180604+0000","last_clean":"2024-04-26T22:47:43.180604+0000","last_became_active":"2024-04-26T22:47:41.781111+0000","last_became_peered":"2024-04-26T22:47:41.781111+0000","last_unstale":"2024-04-26T22:47:43.180604+0000","last_undegraded":"2024-04-26T22:47:43.180604+0000","last_fullsized":"2024-04-26T22:47:43.180604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:54.930233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452656+0000","last_change":"2024-04-26T22:47:41.782289+0000","last_active":"2024-04-26T22:47:43.452656+0000","last_peered":"2024-04-26T22:47:43.452656+0000","last_clean":"2024-04-26T22:47:43.452656+0000","last_became_active":"2024-04-26T22:47:41.781922+0000","last_became_peered":"2024-04-26T22:47:41.781922+0000","last_unstale":"2024-04-26T22:47:43.452656+0000","last_undegraded":"2024-04-26T22:47:43.452656+0000","last_fullsized":"2024-04-26T22:47:43.452656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:30.146485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767338+0000","last_change":"2024-04-26T22:47:41.770970+0000","last_active":"2024-04-26T22:47:42.767338+0000","last_peered":"2024-04-26T22:47:42.767338+0000","last_clean":"2024-04-26T22:47:42.767338+0000","last_became_active":"2024-04-26T22:47:41.770502+0000","last_became_peered":"2024-04-26T22:47:41.770502+0000","last_unstale":"2024-04-26T22:47:42.767338+0000","last_undegraded":"2024-04-26T22:47:42.767338+0000","last_fullsized":"2024-04-26T22:47:42.767338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.988613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451533+0000","last_change":"2024-04-26T22:47:41.779941+0000","last_active":"2024-04-26T22:47:43.451533+0000","last_peered":"2024-04-26T22:47:43.451533+0000","last_clean":"2024-04-26T22:47:43.451533+0000","last_became_active":"2024-04-26T22:47:41.779592+0000","last_became_peered":"2024-04-26T22:47:41.779592+0000","last_unstale":"2024-04-26T22:47:43.451533+0000","last_undegraded":"2024-04-26T22:47:43.451533+0000","last_fullsized":"2024-04-26T22:47:43.451533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:58.628553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451337+0000","last_change":"2024-04-26T22:47:41.781480+0000","last_active":"2024-04-26T22:47:43.451337+0000","last_peered":"2024-04-26T22:47:43.451337+0000","last_clean":"2024-04-26T22:47:43.451337+0000","last_became_active":"2024-04-26T22:47:41.781176+0000","last_became_peered":"2024-04-26T22:47:41.781176+0000","last_unstale":"2024-04-26T22:47:43.451337+0000","last_undegraded":"2024-04-26T22:47:43.451337+0000","last_fullsized":"2024-04-26T22:47:43.451337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:26.361293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451428+0000","last_change":"2024-04-26T22:47:41.771277+0000","last_active":"2024-04-26T22:47:43.451428+0000","last_peered":"2024-04-26T22:47:43.451428+0000","last_clean":"2024-04-26T22:47:43.451428+0000","last_became_active":"2024-04-26T22:47:41.770933+0000","last_became_peered":"2024-04-26T22:47:41.770933+0000","last_unstale":"2024-04-26T22:47:43.451428+0000","last_undegraded":"2024-04-26T22:47:43.451428+0000","last_fullsized":"2024-04-26T22:47:43.451428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:15.045678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.780764+0000","last_change":"2024-04-26T22:47:41.778656+0000","last_active":"2024-04-26T22:47:41.780764+0000","last_peered":"2024-04-26T22:47:41.780764+0000","last_clean":"2024-04-26T22:47:41.780764+0000","last_became_active":"2024-04-26T22:47:41.778282+0000","last_became_peered":"2024-04-26T22:47:41.778282+0000","last_unstale":"2024-04-26T22:47:41.780764+0000","last_undegraded":"2024-04-26T22:47:41.780764+0000","last_fullsized":"2024-04-26T22:47:41.780764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:37.430803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452473+0000","last_change":"2024-04-26T22:47:41.776965+0000","last_active":"2024-04-26T22:47:43.452473+0000","last_peered":"2024-04-26T22:47:43.452473+0000","last_clean":"2024-04-26T22:47:43.452473+0000","last_became_active":"2024-04-26T22:47:41.776625+0000","last_became_peered":"2024-04-26T22:47:41.776625+0000","last_unstale":"2024-04-26T22:47:43.452473+0000","last_undegraded":"2024-04-26T22:47:43.452473+0000","last_fullsized":"2024-04-26T22:47:43.452473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:38.601785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770568+0000","last_change":"2024-04-26T22:47:41.796742+0000","last_active":"2024-04-26T22:47:42.770568+0000","last_peered":"2024-04-26T22:47:42.770568+0000","last_clean":"2024-04-26T22:47:42.770568+0000","last_became_active":"2024-04-26T22:47:41.796362+0000","last_became_peered":"2024-04-26T22:47:41.796362+0000","last_unstale":"2024-04-26T22:47:42.770568+0000","last_undegraded":"2024-04-26T22:47:42.770568+0000","last_fullsized":"2024-04-26T22:47:42.770568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:01:55.092075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.824264+0000","last_change":"2024-04-26T22:47:40.771619+0000","last_active":"2024-04-26T22:47:44.824264+0000","last_peered":"2024-04-26T22:47:44.824264+0000","last_clean":"2024-04-26T22:47:44.824264+0000","last_became_active":"2024-04-26T22:47:40.771262+0000","last_became_peered":"2024-04-26T22:47:40.771262+0000","last_unstale":"2024-04-26T22:47:44.824264+0000","last_undegraded":"2024-04-26T22:47:44.824264+0000","last_fullsized":"2024-04-26T22:47:44.824264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:19.528487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452747+0000","last_change":"2024-04-26T22:47:41.784162+0000","last_active":"2024-04-26T22:47:43.452747+0000","last_peered":"2024-04-26T22:47:43.452747+0000","last_clean":"2024-04-26T22:47:43.452747+0000","last_became_active":"2024-04-26T22:47:41.783821+0000","last_became_peered":"2024-04-26T22:47:41.783821+0000","last_unstale":"2024-04-26T22:47:43.452747+0000","last_undegraded":"2024-04-26T22:47:43.452747+0000","last_fullsized":"2024-04-26T22:47:43.452747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:17:04.237927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452700+0000","last_change":"2024-04-26T22:47:40.779858+0000","last_active":"2024-04-26T22:47:43.452700+0000","last_peered":"2024-04-26T22:47:43.452700+0000","last_clean":"2024-04-26T22:47:43.452700+0000","last_became_active":"2024-04-26T22:47:40.779550+0000","last_became_peered":"2024-04-26T22:47:40.779550+0000","last_unstale":"2024-04-26T22:47:43.452700+0000","last_undegraded":"2024-04-26T22:47:43.452700+0000","last_fullsized":"2024-04-26T22:47:43.452700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:40:54.056426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451574+0000","last_change":"2024-04-26T22:47:41.769865+0000","last_active":"2024-04-26T22:47:43.451574+0000","last_peered":"2024-04-26T22:47:43.451574+0000","last_clean":"2024-04-26T22:47:43.451574+0000","last_became_active":"2024-04-26T22:47:41.769545+0000","last_became_peered":"2024-04-26T22:47:41.769545+0000","last_unstale":"2024-04-26T22:47:43.451574+0000","last_undegraded":"2024-04-26T22:47:43.451574+0000","last_fullsized":"2024-04-26T22:47:43.451574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:39.715241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'11","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.808555+0000","last_change":"2024-04-26T22:47:40.771234+0000","last_active":"2024-04-26T22:47:46.808555+0000","last_peered":"2024-04-26T22:47:46.808555+0000","last_clean":"2024-04-26T22:47:46.808555+0000","last_became_active":"2024-04-26T22:47:40.770876+0000","last_became_peered":"2024-04-26T22:47:40.770876+0000","last_unstale":"2024-04-26T22:47:46.808555+0000","last_undegraded":"2024-04-26T22:47:46.808555+0000","last_fullsized":"2024-04-26T22:47:46.808555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:50:41.272029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451630+0000","last_change":"2024-04-26T22:47:41.770470+0000","last_active":"2024-04-26T22:47:43.451630+0000","last_peered":"2024-04-26T22:47:43.451630+0000","last_clean":"2024-04-26T22:47:43.451630+0000","last_became_active":"2024-04-26T22:47:41.767318+0000","last_became_peered":"2024-04-26T22:47:41.767318+0000","last_unstale":"2024-04-26T22:47:43.451630+0000","last_undegraded":"2024-04-26T22:47:43.451630+0000","last_fullsized":"2024-04-26T22:47:43.451630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:45.324118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.807285+0000","last_change":"2024-04-26T22:47:40.775811+0000","last_active":"2024-04-26T22:47:46.807285+0000","last_peered":"2024-04-26T22:47:46.807285+0000","last_clean":"2024-04-26T22:47:46.807285+0000","last_became_active":"2024-04-26T22:47:40.775349+0000","last_became_peered":"2024-04-26T22:47:40.775349+0000","last_unstale":"2024-04-26T22:47:46.807285+0000","last_undegraded":"2024-04-26T22:47:46.807285+0000","last_fullsized":"2024-04-26T22:47:46.807285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453439+0000","last_change":"2024-04-26T22:47:41.778304+0000","last_active":"2024-04-26T22:47:43.453439+0000","last_peered":"2024-04-26T22:47:43.453439+0000","last_clean":"2024-04-26T22:47:43.453439+0000","last_became_active":"2024-04-26T22:47:41.777930+0000","last_became_peered":"2024-04-26T22:47:41.777930+0000","last_unstale":"2024-04-26T22:47:43.453439+0000","last_undegraded":"2024-04-26T22:47:43.453439+0000","last_fullsized":"2024-04-26T22:47:43.453439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:04.366686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819169+0000","last_change":"2024-04-26T22:47:40.774855+0000","last_active":"2024-04-26T22:47:43.819169+0000","last_peered":"2024-04-26T22:47:43.819169+0000","last_clean":"2024-04-26T22:47:43.819169+0000","last_became_active":"2024-04-26T22:47:40.774517+0000","last_became_peered":"2024-04-26T22:47:40.774517+0000","last_unstale":"2024-04-26T22:47:43.819169+0000","last_undegraded":"2024-04-26T22:47:43.819169+0000","last_fullsized":"2024-04-26T22:47:43.819169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.811072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452357+0000","last_change":"2024-04-26T22:47:41.781432+0000","last_active":"2024-04-26T22:47:43.452357+0000","last_peered":"2024-04-26T22:47:43.452357+0000","last_clean":"2024-04-26T22:47:43.452357+0000","last_became_active":"2024-04-26T22:47:41.781035+0000","last_became_peered":"2024-04-26T22:47:41.781035+0000","last_unstale":"2024-04-26T22:47:43.452357+0000","last_undegraded":"2024-04-26T22:47:43.452357+0000","last_fullsized":"2024-04-26T22:47:43.452357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:31.730087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:45.833589+0000","last_change":"2024-04-26T22:47:40.770419+0000","last_active":"2024-04-26T22:47:45.833589+0000","last_peered":"2024-04-26T22:47:45.833589+0000","last_clean":"2024-04-26T22:47:45.833589+0000","last_became_active":"2024-04-26T22:47:40.769923+0000","last_became_peered":"2024-04-26T22:47:40.769923+0000","last_unstale":"2024-04-26T22:47:45.833589+0000","last_undegraded":"2024-04-26T22:47:45.833589+0000","last_fullsized":"2024-04-26T22:47:45.833589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:15.618904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453022+0000","last_change":"2024-04-26T22:47:41.778334+0000","last_active":"2024-04-26T22:47:43.453022+0000","last_peered":"2024-04-26T22:47:43.453022+0000","last_clean":"2024-04-26T22:47:43.453022+0000","last_became_active":"2024-04-26T22:47:41.777822+0000","last_became_peered":"2024-04-26T22:47:41.777822+0000","last_unstale":"2024-04-26T22:47:43.453022+0000","last_undegraded":"2024-04-26T22:47:43.453022+0000","last_fullsized":"2024-04-26T22:47:43.453022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:57.828165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.823041+0000","last_change":"2024-04-26T22:47:40.779524+0000","last_active":"2024-04-26T22:47:44.823041+0000","last_peered":"2024-04-26T22:47:44.823041+0000","last_clean":"2024-04-26T22:47:44.823041+0000","last_became_active":"2024-04-26T22:47:40.778998+0000","last_became_peered":"2024-04-26T22:47:40.778998+0000","last_unstale":"2024-04-26T22:47:44.823041+0000","last_undegraded":"2024-04-26T22:47:44.823041+0000","last_fullsized":"2024-04-26T22:47:44.823041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:59.979648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451432+0000","last_change":"2024-04-26T22:47:41.768883+0000","last_active":"2024-04-26T22:47:43.451432+0000","last_peered":"2024-04-26T22:47:43.451432+0000","last_clean":"2024-04-26T22:47:43.451432+0000","last_became_active":"2024-04-26T22:47:41.768577+0000","last_became_peered":"2024-04-26T22:47:41.768577+0000","last_unstale":"2024-04-26T22:47:43.451432+0000","last_undegraded":"2024-04-26T22:47:43.451432+0000","last_fullsized":"2024-04-26T22:47:43.451432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:29.267873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452278+0000","last_change":"2024-04-26T22:47:40.774688+0000","last_active":"2024-04-26T22:47:43.452278+0000","last_peered":"2024-04-26T22:47:43.452278+0000","last_clean":"2024-04-26T22:47:43.452278+0000","last_became_active":"2024-04-26T22:47:40.774312+0000","last_became_peered":"2024-04-26T22:47:40.774312+0000","last_unstale":"2024-04-26T22:47:43.452278+0000","last_undegraded":"2024-04-26T22:47:43.452278+0000","last_fullsized":"2024-04-26T22:47:43.452278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:03:34.797965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452470+0000","last_change":"2024-04-26T22:47:41.776948+0000","last_active":"2024-04-26T22:47:43.452470+0000","last_peered":"2024-04-26T22:47:43.452470+0000","last_clean":"2024-04-26T22:47:43.452470+0000","last_became_active":"2024-04-26T22:47:41.776583+0000","last_became_peered":"2024-04-26T22:47:41.776583+0000","last_unstale":"2024-04-26T22:47:43.452470+0000","last_undegraded":"2024-04-26T22:47:43.452470+0000","last_fullsized":"2024-04-26T22:47:43.452470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:42.687249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819273+0000","last_change":"2024-04-26T22:47:40.772060+0000","last_active":"2024-04-26T22:47:43.819273+0000","last_peered":"2024-04-26T22:47:43.819273+0000","last_clean":"2024-04-26T22:47:43.819273+0000","last_became_active":"2024-04-26T22:47:40.771690+0000","last_became_peered":"2024-04-26T22:47:40.771690+0000","last_unstale":"2024-04-26T22:47:43.819273+0000","last_undegraded":"2024-04-26T22:47:43.819273+0000","last_fullsized":"2024-04-26T22:47:43.819273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:53.619399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451834+0000","last_change":"2024-04-26T22:47:41.776902+0000","last_active":"2024-04-26T22:47:43.451834+0000","last_peered":"2024-04-26T22:47:43.451834+0000","last_clean":"2024-04-26T22:47:43.451834+0000","last_became_active":"2024-04-26T22:47:41.776475+0000","last_became_peered":"2024-04-26T22:47:41.776475+0000","last_unstale":"2024-04-26T22:47:43.451834+0000","last_undegraded":"2024-04-26T22:47:43.451834+0000","last_fullsized":"2024-04-26T22:47:43.451834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:57:17.674715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.810754+0000","last_change":"2024-04-26T22:47:40.775828+0000","last_active":"2024-04-26T22:47:46.810754+0000","last_peered":"2024-04-26T22:47:46.810754+0000","last_clean":"2024-04-26T22:47:46.810754+0000","last_became_active":"2024-04-26T22:47:40.775439+0000","last_became_peered":"2024-04-26T22:47:40.775439+0000","last_unstale":"2024-04-26T22:47:46.810754+0000","last_undegraded":"2024-04-26T22:47:46.810754+0000","last_fullsized":"2024-04-26T22:47:46.810754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:28:21.061843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451452+0000","last_change":"2024-04-26T22:47:41.782731+0000","last_active":"2024-04-26T22:47:43.451452+0000","last_peered":"2024-04-26T22:47:43.451452+0000","last_clean":"2024-04-26T22:47:43.451452+0000","last_became_active":"2024-04-26T22:47:41.781390+0000","last_became_peered":"2024-04-26T22:47:41.781390+0000","last_unstale":"2024-04-26T22:47:43.451452+0000","last_undegraded":"2024-04-26T22:47:43.451452+0000","last_fullsized":"2024-04-26T22:47:43.451452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:15.349252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453381+0000","last_change":"2024-04-26T22:47:40.770488+0000","last_active":"2024-04-26T22:47:43.453381+0000","last_peered":"2024-04-26T22:47:43.453381+0000","last_clean":"2024-04-26T22:47:43.453381+0000","last_became_active":"2024-04-26T22:47:40.770049+0000","last_became_peered":"2024-04-26T22:47:40.770049+0000","last_unstale":"2024-04-26T22:47:43.453381+0000","last_undegraded":"2024-04-26T22:47:43.453381+0000","last_fullsized":"2024-04-26T22:47:43.453381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:03.391917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451889+0000","last_change":"2024-04-26T22:47:41.778433+0000","last_active":"2024-04-26T22:47:43.451889+0000","last_peered":"2024-04-26T22:47:43.451889+0000","last_clean":"2024-04-26T22:47:43.451889+0000","last_became_active":"2024-04-26T22:47:41.777603+0000","last_became_peered":"2024-04-26T22:47:41.777603+0000","last_unstale":"2024-04-26T22:47:43.451889+0000","last_undegraded":"2024-04-26T22:47:43.451889+0000","last_fullsized":"2024-04-26T22:47:43.451889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:02.206291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.816917+0000","last_change":"2024-04-26T22:47:40.774417+0000","last_active":"2024-04-26T22:47:43.816917+0000","last_peered":"2024-04-26T22:47:43.816917+0000","last_clean":"2024-04-26T22:47:43.816917+0000","last_became_active":"2024-04-26T22:47:40.773932+0000","last_became_peered":"2024-04-26T22:47:40.773932+0000","last_unstale":"2024-04-26T22:47:43.816917+0000","last_undegraded":"2024-04-26T22:47:43.816917+0000","last_fullsized":"2024-04-26T22:47:43.816917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:15.694401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452087+0000","last_change":"2024-04-26T22:47:41.787466+0000","last_active":"2024-04-26T22:47:43.452087+0000","last_peered":"2024-04-26T22:47:43.452087+0000","last_clean":"2024-04-26T22:47:43.452087+0000","last_became_active":"2024-04-26T22:47:41.787105+0000","last_became_peered":"2024-04-26T22:47:41.787105+0000","last_unstale":"2024-04-26T22:47:43.452087+0000","last_undegraded":"2024-04-26T22:47:43.452087+0000","last_fullsized":"2024-04-26T22:47:43.452087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:48:34.656495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.823363+0000","last_change":"2024-04-26T22:47:40.767810+0000","last_active":"2024-04-26T22:47:44.823363+0000","last_peered":"2024-04-26T22:47:44.823363+0000","last_clean":"2024-04-26T22:47:44.823363+0000","last_became_active":"2024-04-26T22:47:40.767508+0000","last_became_peered":"2024-04-26T22:47:40.767508+0000","last_unstale":"2024-04-26T22:47:44.823363+0000","last_undegraded":"2024-04-26T22:47:44.823363+0000","last_fullsized":"2024-04-26T22:47:44.823363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:32.023939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767072+0000","last_change":"2024-04-26T22:47:41.774134+0000","last_active":"2024-04-26T22:47:42.767072+0000","last_peered":"2024-04-26T22:47:42.767072+0000","last_clean":"2024-04-26T22:47:42.767072+0000","last_became_active":"2024-04-26T22:47:41.773613+0000","last_became_peered":"2024-04-26T22:47:41.773613+0000","last_unstale":"2024-04-26T22:47:42.767072+0000","last_undegraded":"2024-04-26T22:47:42.767072+0000","last_fullsized":"2024-04-26T22:47:42.767072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:41:51.317633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.817168+0000","last_change":"2024-04-26T22:47:40.776836+0000","last_active":"2024-04-26T22:47:43.817168+0000","last_peered":"2024-04-26T22:47:43.817168+0000","last_clean":"2024-04-26T22:47:43.817168+0000","last_became_active":"2024-04-26T22:47:40.776493+0000","last_became_peered":"2024-04-26T22:47:40.776493+0000","last_unstale":"2024-04-26T22:47:43.817168+0000","last_undegraded":"2024-04-26T22:47:43.817168+0000","last_fullsized":"2024-04-26T22:47:43.817168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:55.695456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180520+0000","last_change":"2024-04-26T22:47:41.784562+0000","last_active":"2024-04-26T22:47:43.180520+0000","last_peered":"2024-04-26T22:47:43.180520+0000","last_clean":"2024-04-26T22:47:43.180520+0000","last_became_active":"2024-04-26T22:47:41.784150+0000","last_became_peered":"2024-04-26T22:47:41.784150+0000","last_unstale":"2024-04-26T22:47:43.180520+0000","last_undegraded":"2024-04-26T22:47:43.180520+0000","last_fullsized":"2024-04-26T22:47:43.180520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:35.247848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452024+0000","last_change":"2024-04-26T22:47:40.768984+0000","last_active":"2024-04-26T22:47:43.452024+0000","last_peered":"2024-04-26T22:47:43.452024+0000","last_clean":"2024-04-26T22:47:43.452024+0000","last_became_active":"2024-04-26T22:47:40.768596+0000","last_became_peered":"2024-04-26T22:47:40.768596+0000","last_unstale":"2024-04-26T22:47:43.452024+0000","last_undegraded":"2024-04-26T22:47:43.452024+0000","last_fullsized":"2024-04-26T22:47:43.452024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:44.099790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180027+0000","last_change":"2024-04-26T22:47:41.778100+0000","last_active":"2024-04-26T22:47:43.180027+0000","last_peered":"2024-04-26T22:47:43.180027+0000","last_clean":"2024-04-26T22:47:43.180027+0000","last_became_active":"2024-04-26T22:47:41.777775+0000","last_became_peered":"2024-04-26T22:47:41.777775+0000","last_unstale":"2024-04-26T22:47:43.180027+0000","last_undegraded":"2024-04-26T22:47:43.180027+0000","last_fullsized":"2024-04-26T22:47:43.180027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:52.698089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.798848+0000","last_change":"2024-04-26T22:47:40.761904+0000","last_active":"2024-04-26T22:47:46.798848+0000","last_peered":"2024-04-26T22:47:46.798848+0000","last_clean":"2024-04-26T22:47:46.798848+0000","last_became_active":"2024-04-26T22:47:40.761591+0000","last_became_peered":"2024-04-26T22:47:40.761591+0000","last_unstale":"2024-04-26T22:47:46.798848+0000","last_undegraded":"2024-04-26T22:47:46.798848+0000","last_fullsized":"2024-04-26T22:47:46.798848+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:55.774669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452047+0000","last_change":"2024-04-26T22:47:41.780703+0000","last_active":"2024-04-26T22:47:43.452047+0000","last_peered":"2024-04-26T22:47:43.452047+0000","last_clean":"2024-04-26T22:47:43.452047+0000","last_became_active":"2024-04-26T22:47:41.775023+0000","last_became_peered":"2024-04-26T22:47:41.775023+0000","last_unstale":"2024-04-26T22:47:43.452047+0000","last_undegraded":"2024-04-26T22:47:43.452047+0000","last_fullsized":"2024-04-26T22:47:43.452047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:09.515514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.799020+0000","last_change":"2024-04-26T22:47:40.766095+0000","last_active":"2024-04-26T22:47:46.799020+0000","last_peered":"2024-04-26T22:47:46.799020+0000","last_clean":"2024-04-26T22:47:46.799020+0000","last_became_active":"2024-04-26T22:47:40.765751+0000","last_became_peered":"2024-04-26T22:47:40.765751+0000","last_unstale":"2024-04-26T22:47:46.799020+0000","last_undegraded":"2024-04-26T22:47:46.799020+0000","last_fullsized":"2024-04-26T22:47:46.799020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:22.877861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.786084+0000","last_change":"2024-04-26T22:47:41.781793+0000","last_active":"2024-04-26T22:47:41.786084+0000","last_peered":"2024-04-26T22:47:41.786084+0000","last_clean":"2024-04-26T22:47:41.786084+0000","last_became_active":"2024-04-26T22:47:41.781470+0000","last_became_peered":"2024-04-26T22:47:41.781470+0000","last_unstale":"2024-04-26T22:47:41.786084+0000","last_undegraded":"2024-04-26T22:47:41.786084+0000","last_fullsized":"2024-04-26T22:47:41.786084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:00.668938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452183+0000","last_change":"2024-04-26T22:47:40.767364+0000","last_active":"2024-04-26T22:47:43.452183+0000","last_peered":"2024-04-26T22:47:43.452183+0000","last_clean":"2024-04-26T22:47:43.452183+0000","last_became_active":"2024-04-26T22:47:40.766907+0000","last_became_peered":"2024-04-26T22:47:40.766907+0000","last_unstale":"2024-04-26T22:47:43.452183+0000","last_undegraded":"2024-04-26T22:47:43.452183+0000","last_fullsized":"2024-04-26T22:47:43.452183+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:56.408996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.768356+0000","last_change":"2024-04-26T22:47:41.784095+0000","last_active":"2024-04-26T22:47:42.768356+0000","last_peered":"2024-04-26T22:47:42.768356+0000","last_clean":"2024-04-26T22:47:42.768356+0000","last_became_active":"2024-04-26T22:47:41.783749+0000","last_became_peered":"2024-04-26T22:47:41.783749+0000","last_unstale":"2024-04-26T22:47:42.768356+0000","last_undegraded":"2024-04-26T22:47:42.768356+0000","last_fullsized":"2024-04-26T22:47:42.768356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:11:26.530223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.811482+0000","last_change":"2024-04-26T22:47:40.778988+0000","last_active":"2024-04-26T22:47:46.811482+0000","last_peered":"2024-04-26T22:47:46.811482+0000","last_clean":"2024-04-26T22:47:46.811482+0000","last_became_active":"2024-04-26T22:47:40.778686+0000","last_became_peered":"2024-04-26T22:47:40.778686+0000","last_unstale":"2024-04-26T22:47:46.811482+0000","last_undegraded":"2024-04-26T22:47:46.811482+0000","last_fullsized":"2024-04-26T22:47:46.811482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.525866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767942+0000","last_change":"2024-04-26T22:47:41.787168+0000","last_active":"2024-04-26T22:47:42.767942+0000","last_peered":"2024-04-26T22:47:42.767942+0000","last_clean":"2024-04-26T22:47:42.767942+0000","last_became_active":"2024-04-26T22:47:41.786811+0000","last_became_peered":"2024-04-26T22:47:41.786811+0000","last_unstale":"2024-04-26T22:47:42.767942+0000","last_undegraded":"2024-04-26T22:47:42.767942+0000","last_fullsized":"2024-04-26T22:47:42.767942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:34.578058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452435+0000","last_change":"2024-04-26T22:47:41.780945+0000","last_active":"2024-04-26T22:47:43.452435+0000","last_peered":"2024-04-26T22:47:43.452435+0000","last_clean":"2024-04-26T22:47:43.452435+0000","last_became_active":"2024-04-26T22:47:41.779325+0000","last_became_peered":"2024-04-26T22:47:41.779325+0000","last_unstale":"2024-04-26T22:47:43.452435+0000","last_undegraded":"2024-04-26T22:47:43.452435+0000","last_fullsized":"2024-04-26T22:47:43.452435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:57.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.824536+0000","last_change":"2024-04-26T22:47:40.781706+0000","last_active":"2024-04-26T22:47:44.824536+0000","last_peered":"2024-04-26T22:47:44.824536+0000","last_clean":"2024-04-26T22:47:44.824536+0000","last_became_active":"2024-04-26T22:47:40.781014+0000","last_became_peered":"2024-04-26T22:47:40.781014+0000","last_unstale":"2024-04-26T22:47:44.824536+0000","last_undegraded":"2024-04-26T22:47:44.824536+0000","last_fullsized":"2024-04-26T22:47:44.824536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:17:16.523767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452727+0000","last_change":"2024-04-26T22:47:41.772852+0000","last_active":"2024-04-26T22:47:43.452727+0000","last_peered":"2024-04-26T22:47:43.452727+0000","last_clean":"2024-04-26T22:47:43.452727+0000","last_became_active":"2024-04-26T22:47:41.772502+0000","last_became_peered":"2024-04-26T22:47:41.772502+0000","last_unstale":"2024-04-26T22:47:43.452727+0000","last_undegraded":"2024-04-26T22:47:43.452727+0000","last_fullsized":"2024-04-26T22:47:43.452727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:36.870593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819431+0000","last_change":"2024-04-26T22:47:40.768151+0000","last_active":"2024-04-26T22:47:43.819431+0000","last_peered":"2024-04-26T22:47:43.819431+0000","last_clean":"2024-04-26T22:47:43.819431+0000","last_became_active":"2024-04-26T22:47:40.767756+0000","last_became_peered":"2024-04-26T22:47:40.767756+0000","last_unstale":"2024-04-26T22:47:43.819431+0000","last_undegraded":"2024-04-26T22:47:43.819431+0000","last_fullsized":"2024-04-26T22:47:43.819431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:38:09.059145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":337,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180017+0000","last_change":"2024-04-26T22:46:24.327741+0000","last_active":"2024-04-26T22:47:43.180017+0000","last_peered":"2024-04-26T22:47:43.180017+0000","last_clean":"2024-04-26T22:47:43.180017+0000","last_became_active":"2024-04-26T22:46:24.327280+0000","last_became_peered":"2024-04-26T22:46:24.327280+0000","last_unstale":"2024-04-26T22:47:43.180017+0000","last_undegraded":"2024-04-26T22:47:43.180017+0000","last_fullsized":"2024-04-26T22:47:43.180017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_clean_scrub_stamp":"2024-04-26T22:42:55.984240+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:47.094131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.766860+0000","last_change":"2024-04-26T22:47:41.779195+0000","last_active":"2024-04-26T22:47:42.766860+0000","last_peered":"2024-04-26T22:47:42.766860+0000","last_clean":"2024-04-26T22:47:42.766860+0000","last_became_active":"2024-04-26T22:47:41.778682+0000","last_became_peered":"2024-04-26T22:47:41.778682+0000","last_unstale":"2024-04-26T22:47:42.766860+0000","last_undegraded":"2024-04-26T22:47:42.766860+0000","last_fullsized":"2024-04-26T22:47:42.766860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:37.983340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.821930+0000","last_change":"2024-04-26T22:47:40.774817+0000","last_active":"2024-04-26T22:47:44.821930+0000","last_peered":"2024-04-26T22:47:44.821930+0000","last_clean":"2024-04-26T22:47:44.821930+0000","last_became_active":"2024-04-26T22:47:40.772660+0000","last_became_peered":"2024-04-26T22:47:40.772660+0000","last_unstale":"2024-04-26T22:47:44.821930+0000","last_undegraded":"2024-04-26T22:47:44.821930+0000","last_fullsized":"2024-04-26T22:47:44.821930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:30.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.795377+0000","last_change":"2024-04-26T22:47:40.773595+0000","last_active":"2024-04-26T22:47:46.795377+0000","last_peered":"2024-04-26T22:47:46.795377+0000","last_clean":"2024-04-26T22:47:46.795377+0000","last_became_active":"2024-04-26T22:47:40.773199+0000","last_became_peered":"2024-04-26T22:47:40.773199+0000","last_unstale":"2024-04-26T22:47:46.795377+0000","last_undegraded":"2024-04-26T22:47:46.795377+0000","last_fullsized":"2024-04-26T22:47:46.795377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:19:31.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767273+0000","last_change":"2024-04-26T22:47:41.780428+0000","last_active":"2024-04-26T22:47:42.767273+0000","last_peered":"2024-04-26T22:47:42.767273+0000","last_clean":"2024-04-26T22:47:42.767273+0000","last_became_active":"2024-04-26T22:47:41.779468+0000","last_became_peered":"2024-04-26T22:47:41.779468+0000","last_unstale":"2024-04-26T22:47:42.767273+0000","last_undegraded":"2024-04-26T22:47:42.767273+0000","last_fullsized":"2024-04-26T22:47:42.767273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:05.067347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.823351+0000","last_change":"2024-04-26T22:47:40.770624+0000","last_active":"2024-04-26T22:47:44.823351+0000","last_peered":"2024-04-26T22:47:44.823351+0000","last_clean":"2024-04-26T22:47:44.823351+0000","last_became_active":"2024-04-26T22:47:40.770250+0000","last_became_peered":"2024-04-26T22:47:40.770250+0000","last_unstale":"2024-04-26T22:47:44.823351+0000","last_undegraded":"2024-04-26T22:47:44.823351+0000","last_fullsized":"2024-04-26T22:47:44.823351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:16:53.527984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452706+0000","last_change":"2024-04-26T22:47:41.773667+0000","last_active":"2024-04-26T22:47:43.452706+0000","last_peered":"2024-04-26T22:47:43.452706+0000","last_clean":"2024-04-26T22:47:43.452706+0000","last_became_active":"2024-04-26T22:47:41.772876+0000","last_became_peered":"2024-04-26T22:47:41.772876+0000","last_unstale":"2024-04-26T22:47:43.452706+0000","last_undegraded":"2024-04-26T22:47:43.452706+0000","last_fullsized":"2024-04-26T22:47:43.452706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.363992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.818808+0000","last_change":"2024-04-26T22:47:40.774090+0000","last_active":"2024-04-26T22:47:43.818808+0000","last_peered":"2024-04-26T22:47:43.818808+0000","last_clean":"2024-04-26T22:47:43.818808+0000","last_became_active":"2024-04-26T22:47:40.773543+0000","last_became_peered":"2024-04-26T22:47:40.773543+0000","last_unstale":"2024-04-26T22:47:43.818808+0000","last_undegraded":"2024-04-26T22:47:43.818808+0000","last_fullsized":"2024-04-26T22:47:43.818808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:17:06.907699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452376+0000","last_change":"2024-04-26T22:47:41.781231+0000","last_active":"2024-04-26T22:47:43.452376+0000","last_peered":"2024-04-26T22:47:43.452376+0000","last_clean":"2024-04-26T22:47:43.452376+0000","last_became_active":"2024-04-26T22:47:41.780837+0000","last_became_peered":"2024-04-26T22:47:41.780837+0000","last_unstale":"2024-04-26T22:47:43.452376+0000","last_undegraded":"2024-04-26T22:47:43.452376+0000","last_fullsized":"2024-04-26T22:47:43.452376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:24.491628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.818199+0000","last_change":"2024-04-26T22:47:40.771454+0000","last_active":"2024-04-26T22:47:43.818199+0000","last_peered":"2024-04-26T22:47:43.818199+0000","last_clean":"2024-04-26T22:47:43.818199+0000","last_became_active":"2024-04-26T22:47:40.770488+0000","last_became_peered":"2024-04-26T22:47:40.770488+0000","last_unstale":"2024-04-26T22:47:43.818199+0000","last_undegraded":"2024-04-26T22:47:43.818199+0000","last_fullsized":"2024-04-26T22:47:43.818199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:19.422331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451669+0000","last_change":"2024-04-26T22:47:41.776683+0000","last_active":"2024-04-26T22:47:43.451669+0000","last_peered":"2024-04-26T22:47:43.451669+0000","last_clean":"2024-04-26T22:47:43.451669+0000","last_became_active":"2024-04-26T22:47:41.776309+0000","last_became_peered":"2024-04-26T22:47:41.776309+0000","last_unstale":"2024-04-26T22:47:43.451669+0000","last_undegraded":"2024-04-26T22:47:43.451669+0000","last_fullsized":"2024-04-26T22:47:43.451669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:45.604270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.818910+0000","last_change":"2024-04-26T22:47:40.774392+0000","last_active":"2024-04-26T22:47:43.818910+0000","last_peered":"2024-04-26T22:47:43.818910+0000","last_clean":"2024-04-26T22:47:43.818910+0000","last_became_active":"2024-04-26T22:47:40.773682+0000","last_became_peered":"2024-04-26T22:47:40.773682+0000","last_unstale":"2024-04-26T22:47:43.818910+0000","last_undegraded":"2024-04-26T22:47:43.818910+0000","last_fullsized":"2024-04-26T22:47:43.818910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:06:56.769791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.785944+0000","last_change":"2024-04-26T22:47:41.784191+0000","last_active":"2024-04-26T22:47:41.785944+0000","last_peered":"2024-04-26T22:47:41.785944+0000","last_clean":"2024-04-26T22:47:41.785944+0000","last_became_active":"2024-04-26T22:47:41.783884+0000","last_became_peered":"2024-04-26T22:47:41.783884+0000","last_unstale":"2024-04-26T22:47:41.785944+0000","last_undegraded":"2024-04-26T22:47:41.785944+0000","last_fullsized":"2024-04-26T22:47:41.785944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:50:41.479348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770146+0000","last_change":"2024-04-26T22:47:40.767796+0000","last_active":"2024-04-26T22:47:42.770146+0000","last_peered":"2024-04-26T22:47:42.770146+0000","last_clean":"2024-04-26T22:47:42.770146+0000","last_became_active":"2024-04-26T22:47:40.767450+0000","last_became_peered":"2024-04-26T22:47:40.767450+0000","last_unstale":"2024-04-26T22:47:42.770146+0000","last_undegraded":"2024-04-26T22:47:42.770146+0000","last_fullsized":"2024-04-26T22:47:42.770146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:41.481981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452200+0000","last_change":"2024-04-26T22:47:41.774308+0000","last_active":"2024-04-26T22:47:43.452200+0000","last_peered":"2024-04-26T22:47:43.452200+0000","last_clean":"2024-04-26T22:47:43.452200+0000","last_became_active":"2024-04-26T22:47:41.773713+0000","last_became_peered":"2024-04-26T22:47:41.773713+0000","last_unstale":"2024-04-26T22:47:43.452200+0000","last_undegraded":"2024-04-26T22:47:43.452200+0000","last_fullsized":"2024-04-26T22:47:43.452200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:11.186258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819268+0000","last_change":"2024-04-26T22:47:40.768761+0000","last_active":"2024-04-26T22:47:43.819268+0000","last_peered":"2024-04-26T22:47:43.819268+0000","last_clean":"2024-04-26T22:47:43.819268+0000","last_became_active":"2024-04-26T22:47:40.768434+0000","last_became_peered":"2024-04-26T22:47:40.768434+0000","last_unstale":"2024-04-26T22:47:43.819268+0000","last_undegraded":"2024-04-26T22:47:43.819268+0000","last_fullsized":"2024-04-26T22:47:43.819268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:01.631990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180613+0000","last_change":"2024-04-26T22:47:41.778214+0000","last_active":"2024-04-26T22:47:43.180613+0000","last_peered":"2024-04-26T22:47:43.180613+0000","last_clean":"2024-04-26T22:47:43.180613+0000","last_became_active":"2024-04-26T22:47:41.775889+0000","last_became_peered":"2024-04-26T22:47:41.775889+0000","last_unstale":"2024-04-26T22:47:43.180613+0000","last_undegraded":"2024-04-26T22:47:43.180613+0000","last_fullsized":"2024-04-26T22:47:43.180613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:24.654113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.822543+0000","last_change":"2024-04-26T22:47:40.772366+0000","last_active":"2024-04-26T22:47:44.822543+0000","last_peered":"2024-04-26T22:47:44.822543+0000","last_clean":"2024-04-26T22:47:44.822543+0000","last_became_active":"2024-04-26T22:47:40.772035+0000","last_became_peered":"2024-04-26T22:47:40.772035+0000","last_unstale":"2024-04-26T22:47:44.822543+0000","last_undegraded":"2024-04-26T22:47:44.822543+0000","last_fullsized":"2024-04-26T22:47:44.822543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:54.429132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180656+0000","last_change":"2024-04-26T22:47:41.777074+0000","last_active":"2024-04-26T22:47:43.180656+0000","last_peered":"2024-04-26T22:47:43.180656+0000","last_clean":"2024-04-26T22:47:43.180656+0000","last_became_active":"2024-04-26T22:47:41.776225+0000","last_became_peered":"2024-04-26T22:47:41.776225+0000","last_unstale":"2024-04-26T22:47:43.180656+0000","last_undegraded":"2024-04-26T22:47:43.180656+0000","last_fullsized":"2024-04-26T22:47:43.180656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:06:59.801103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453137+0000","last_change":"2024-04-26T22:47:40.776174+0000","last_active":"2024-04-26T22:47:43.453137+0000","last_peered":"2024-04-26T22:47:43.453137+0000","last_clean":"2024-04-26T22:47:43.453137+0000","last_became_active":"2024-04-26T22:47:40.775866+0000","last_became_peered":"2024-04-26T22:47:40.775866+0000","last_unstale":"2024-04-26T22:47:43.453137+0000","last_undegraded":"2024-04-26T22:47:43.453137+0000","last_fullsized":"2024-04-26T22:47:43.453137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:01.306261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180284+0000","last_change":"2024-04-26T22:47:41.777225+0000","last_active":"2024-04-26T22:47:43.180284+0000","last_peered":"2024-04-26T22:47:43.180284+0000","last_clean":"2024-04-26T22:47:43.180284+0000","last_became_active":"2024-04-26T22:47:41.776814+0000","last_became_peered":"2024-04-26T22:47:41.776814+0000","last_unstale":"2024-04-26T22:47:43.180284+0000","last_undegraded":"2024-04-26T22:47:43.180284+0000","last_fullsized":"2024-04-26T22:47:43.180284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:30.254893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767205+0000","last_change":"2024-04-26T22:47:40.781678+0000","last_active":"2024-04-26T22:47:42.767205+0000","last_peered":"2024-04-26T22:47:42.767205+0000","last_clean":"2024-04-26T22:47:42.767205+0000","last_became_active":"2024-04-26T22:47:40.781156+0000","last_became_peered":"2024-04-26T22:47:40.781156+0000","last_unstale":"2024-04-26T22:47:42.767205+0000","last_undegraded":"2024-04-26T22:47:42.767205+0000","last_fullsized":"2024-04-26T22:47:42.767205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:53.353768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180523+0000","last_change":"2024-04-26T22:47:41.780116+0000","last_active":"2024-04-26T22:47:43.180523+0000","last_peered":"2024-04-26T22:47:43.180523+0000","last_clean":"2024-04-26T22:47:43.180523+0000","last_became_active":"2024-04-26T22:47:41.777443+0000","last_became_peered":"2024-04-26T22:47:41.777443+0000","last_unstale":"2024-04-26T22:47:43.180523+0000","last_undegraded":"2024-04-26T22:47:43.180523+0000","last_fullsized":"2024-04-26T22:47:43.180523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:22.948100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.785849+0000","last_change":"2024-04-26T22:47:41.784618+0000","last_active":"2024-04-26T22:47:41.785849+0000","last_peered":"2024-04-26T22:47:41.785849+0000","last_clean":"2024-04-26T22:47:41.785849+0000","last_became_active":"2024-04-26T22:47:41.784269+0000","last_became_peered":"2024-04-26T22:47:41.784269+0000","last_unstale":"2024-04-26T22:47:41.785849+0000","last_undegraded":"2024-04-26T22:47:41.785849+0000","last_fullsized":"2024-04-26T22:47:41.785849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:33:19.337891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770269+0000","last_change":"2024-04-26T22:47:41.786063+0000","last_active":"2024-04-26T22:47:42.770269+0000","last_peered":"2024-04-26T22:47:42.770269+0000","last_clean":"2024-04-26T22:47:42.770269+0000","last_became_active":"2024-04-26T22:47:41.785657+0000","last_became_peered":"2024-04-26T22:47:41.785657+0000","last_unstale":"2024-04-26T22:47:42.770269+0000","last_undegraded":"2024-04-26T22:47:42.770269+0000","last_fullsized":"2024-04-26T22:47:42.770269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:14.289477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770170+0000","last_change":"2024-04-26T22:47:41.788788+0000","last_active":"2024-04-26T22:47:42.770170+0000","last_peered":"2024-04-26T22:47:42.770170+0000","last_clean":"2024-04-26T22:47:42.770170+0000","last_became_active":"2024-04-26T22:47:41.788434+0000","last_became_peered":"2024-04-26T22:47:41.788434+0000","last_unstale":"2024-04-26T22:47:42.770170+0000","last_undegraded":"2024-04-26T22:47:42.770170+0000","last_fullsized":"2024-04-26T22:47:42.770170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:05.056380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451742+0000","last_change":"2024-04-26T22:47:41.780435+0000","last_active":"2024-04-26T22:47:43.451742+0000","last_peered":"2024-04-26T22:47:43.451742+0000","last_clean":"2024-04-26T22:47:43.451742+0000","last_became_active":"2024-04-26T22:47:41.780126+0000","last_became_peered":"2024-04-26T22:47:41.780126+0000","last_unstale":"2024-04-26T22:47:43.451742+0000","last_undegraded":"2024-04-26T22:47:43.451742+0000","last_fullsized":"2024-04-26T22:47:43.451742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:20.061814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767447+0000","last_change":"2024-04-26T22:47:41.775797+0000","last_active":"2024-04-26T22:47:42.767447+0000","last_peered":"2024-04-26T22:47:42.767447+0000","last_clean":"2024-04-26T22:47:42.767447+0000","last_became_active":"2024-04-26T22:47:41.775286+0000","last_became_peered":"2024-04-26T22:47:41.775286+0000","last_unstale":"2024-04-26T22:47:42.767447+0000","last_undegraded":"2024-04-26T22:47:42.767447+0000","last_fullsized":"2024-04-26T22:47:42.767447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:45.290373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.786030+0000","last_change":"2024-04-26T22:47:41.784714+0000","last_active":"2024-04-26T22:47:41.786030+0000","last_peered":"2024-04-26T22:47:41.786030+0000","last_clean":"2024-04-26T22:47:41.786030+0000","last_became_active":"2024-04-26T22:47:41.783233+0000","last_became_peered":"2024-04-26T22:47:41.783233+0000","last_unstale":"2024-04-26T22:47:41.786030+0000","last_undegraded":"2024-04-26T22:47:41.786030+0000","last_fullsized":"2024-04-26T22:47:41.786030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:24.132003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770703+0000","last_change":"2024-04-26T22:47:41.796781+0000","last_active":"2024-04-26T22:47:42.770703+0000","last_peered":"2024-04-26T22:47:42.770703+0000","last_clean":"2024-04-26T22:47:42.770703+0000","last_became_active":"2024-04-26T22:47:41.796326+0000","last_became_peered":"2024-04-26T22:47:41.796326+0000","last_unstale":"2024-04-26T22:47:42.770703+0000","last_undegraded":"2024-04-26T22:47:42.770703+0000","last_fullsized":"2024-04-26T22:47:42.770703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:30:59.924808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452602+0000","last_change":"2024-04-26T22:47:41.785149+0000","last_active":"2024-04-26T22:47:43.452602+0000","last_peered":"2024-04-26T22:47:43.452602+0000","last_clean":"2024-04-26T22:47:43.452602+0000","last_became_active":"2024-04-26T22:47:41.784807+0000","last_became_peered":"2024-04-26T22:47:41.784807+0000","last_unstale":"2024-04-26T22:47:43.452602+0000","last_undegraded":"2024-04-26T22:47:43.452602+0000","last_fullsized":"2024-04-26T22:47:43.452602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:45.521731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452445+0000","last_change":"2024-04-26T22:47:41.768789+0000","last_active":"2024-04-26T22:47:43.452445+0000","last_peered":"2024-04-26T22:47:43.452445+0000","last_clean":"2024-04-26T22:47:43.452445+0000","last_became_active":"2024-04-26T22:47:41.768466+0000","last_became_peered":"2024-04-26T22:47:41.768466+0000","last_unstale":"2024-04-26T22:47:43.452445+0000","last_undegraded":"2024-04-26T22:47:43.452445+0000","last_fullsized":"2024-04-26T22:47:43.452445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:15.798301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452319+0000","last_change":"2024-04-26T22:47:41.781867+0000","last_active":"2024-04-26T22:47:43.452319+0000","last_peered":"2024-04-26T22:47:43.452319+0000","last_clean":"2024-04-26T22:47:43.452319+0000","last_became_active":"2024-04-26T22:47:41.780648+0000","last_became_peered":"2024-04-26T22:47:41.780648+0000","last_unstale":"2024-04-26T22:47:43.452319+0000","last_undegraded":"2024-04-26T22:47:43.452319+0000","last_fullsized":"2024-04-26T22:47:43.452319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:25.393892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452129+0000","last_change":"2024-04-26T22:47:41.774622+0000","last_active":"2024-04-26T22:47:43.452129+0000","last_peered":"2024-04-26T22:47:43.452129+0000","last_clean":"2024-04-26T22:47:43.452129+0000","last_became_active":"2024-04-26T22:47:41.774262+0000","last_became_peered":"2024-04-26T22:47:41.774262+0000","last_unstale":"2024-04-26T22:47:43.452129+0000","last_undegraded":"2024-04-26T22:47:43.452129+0000","last_fullsized":"2024-04-26T22:47:43.452129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:32.571497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767433+0000","last_change":"2024-04-26T22:47:41.768981+0000","last_active":"2024-04-26T22:47:42.767433+0000","last_peered":"2024-04-26T22:47:42.767433+0000","last_clean":"2024-04-26T22:47:42.767433+0000","last_became_active":"2024-04-26T22:47:41.768515+0000","last_became_peered":"2024-04-26T22:47:41.768515+0000","last_unstale":"2024-04-26T22:47:42.767433+0000","last_undegraded":"2024-04-26T22:47:42.767433+0000","last_fullsized":"2024-04-26T22:47:42.767433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:43:47.680248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.789688+0000","last_change":"2024-04-26T22:47:41.788415+0000","last_active":"2024-04-26T22:47:41.789688+0000","last_peered":"2024-04-26T22:47:41.789688+0000","last_clean":"2024-04-26T22:47:41.789688+0000","last_became_active":"2024-04-26T22:47:41.788076+0000","last_became_peered":"2024-04-26T22:47:41.788076+0000","last_unstale":"2024-04-26T22:47:41.789688+0000","last_undegraded":"2024-04-26T22:47:41.789688+0000","last_fullsized":"2024-04-26T22:47:41.789688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:20.829264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":288005,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5950,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":17603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1032203,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":439305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1032422,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":438633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":440715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":439215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":440789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":3314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T22:47:49.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 -- 172.21.15.1:0/1436839775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608101890 msgr2=0x7f9608101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 --2- 172.21.15.1:0/1436839775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608101890 0x7f9608101cb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9600009ca0 tx=0x7f960002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 -- 172.21.15.1:0/1436839775 shutdown_connections 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 --2- 172.21.15.1:0/1436839775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9608103440 0x7f9608109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 --2- 172.21.15.1:0/1436839775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9608102a80 0x7f9608102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 --2- 172.21.15.1:0/1436839775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608101890 0x7f9608101cb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 -- 172.21.15.1:0/1436839775 >> 172.21.15.1:0/1436839775 conn(0x7f96080fce40 msgr2=0x7f96080ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 -- 172.21.15.1:0/1436839775 shutdown_connections 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 -- 172.21.15.1:0/1436839775 wait complete. 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 Processor -- start 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.427+0000 7f960f07e640 1 -- start start 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9608101890 0x7f9608198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 0x7f9608199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9608103440 0x7f96081a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f960810d410 con 0x7f9608102a80 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f960810d290 con 0x7f9608101890 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f960810d590 con 0x7f9608103440 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 0x7f9608199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 0x7f9608199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36852/0 (socket says 172.21.15.1:36852) 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 -- 172.21.15.1:0/1283942074 learned_addr learned my addr 172.21.15.1:0/1283942074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960ce1a640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9608101890 0x7f9608198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960d61b640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9608103440 0x7f96081a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 -- 172.21.15.1:0/1283942074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9608103440 msgr2=0x7f96081a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9608103440 0x7f96081a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 -- 172.21.15.1:0/1283942074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9608101890 msgr2=0x7f9608198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9608101890 0x7f9608198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 -- 172.21.15.1:0/1283942074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96081a0840 con 0x7f9608102a80 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9607fff640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 0x7f9608199170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f95f800d950 tx=0x7f95f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8017030 con 0x7f9608102a80 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96081a0b30 con 0x7f9608102a80 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95f8004020 con 0x7f9608102a80 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.428+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8005640 con 0x7f9608102a80 2024-04-26T22:47:49.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.429+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96081a10a0 con 0x7f9608102a80 2024-04-26T22:47:49.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.430+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96081a12d0 con 0x7f9608102a80 2024-04-26T22:47:49.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.430+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f95f8009a30 con 0x7f9608102a80 2024-04-26T22:47:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.431+0000 7f9605ffb640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f95f40a3de0 0x7f95f40a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.431+0000 7f960ce1a640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f95f40a3de0 0x7f95f40a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.431+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f95f80cfe40 con 0x7f9608102a80 2024-04-26T22:47:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.432+0000 7f960ce1a640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f95f40a3de0 0x7f95f40a62a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9600000f80 tx=0x7f9600005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.436+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f95f8021020 con 0x7f9608102a80 2024-04-26T22:47:49.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.619+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f96080ff040 con 0x7f95f40a3de0 2024-04-26T22:47:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.626+0000 7f9605ffb640 1 -- 172.21.15.1:0/1283942074 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258935 (secure 0 0 0) 0x7f96080ff040 con 0x7f95f40a3de0 2024-04-26T22:47:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T22:47:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f95f40a3de0 msgr2=0x7f95f40a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f95f40a3de0 0x7f95f40a62a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9600000f80 tx=0x7f9600005f50 comp rx=0 tx=0).stop 2024-04-26T22:47:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 msgr2=0x7f9608199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 0x7f9608199170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f95f800d950 tx=0x7f95f800de10 comp rx=0 tx=0).stop 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 shutdown_connections 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f95f40a3de0 0x7f95f40a62a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9608103440 0x7f96081a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9608102a80 0x7f9608199170 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 --2- 172.21.15.1:0/1283942074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9608101890 0x7f9608198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.629+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 >> 172.21.15.1:0/1283942074 conn(0x7f96080fce40 msgr2=0x7f96080fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.630+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 shutdown_connections 2024-04-26T22:47:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:49.630+0000 7f960f07e640 1 -- 172.21.15.1:0/1283942074 wait complete. 2024-04-26T22:47:49.641 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-26T22:47:47.751007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":133,"num_read_kb":116,"num_write":335,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":269,"ondisk_log_size":269,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":249,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360244,"kb_used_data":11060,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581580,"statfs":{"total":1151940427776,"available":1151571537920,"internally_reserved":0,"allocated":11325440,"data_stored":7040115,"data_compressed":109848,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5950,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.997722"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.780073+0000","last_change":"2024-04-26T22:47:41.777587+0000","last_active":"2024-04-26T22:47:41.780073+0000","last_peered":"2024-04-26T22:47:41.780073+0000","last_clean":"2024-04-26T22:47:41.780073+0000","last_became_active":"2024-04-26T22:47:41.777165+0000","last_became_peered":"2024-04-26T22:47:41.777165+0000","last_unstale":"2024-04-26T22:47:41.780073+0000","last_undegraded":"2024-04-26T22:47:41.780073+0000","last_fullsized":"2024-04-26T22:47:41.780073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:31.561664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451905+0000","last_change":"2024-04-26T22:47:41.776888+0000","last_active":"2024-04-26T22:47:43.451905+0000","last_peered":"2024-04-26T22:47:43.451905+0000","last_clean":"2024-04-26T22:47:43.451905+0000","last_became_active":"2024-04-26T22:47:41.776580+0000","last_became_peered":"2024-04-26T22:47:41.776580+0000","last_unstale":"2024-04-26T22:47:43.451905+0000","last_undegraded":"2024-04-26T22:47:43.451905+0000","last_fullsized":"2024-04-26T22:47:43.451905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:49.263390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451484+0000","last_change":"2024-04-26T22:47:41.784374+0000","last_active":"2024-04-26T22:47:43.451484+0000","last_peered":"2024-04-26T22:47:43.451484+0000","last_clean":"2024-04-26T22:47:43.451484+0000","last_became_active":"2024-04-26T22:47:41.783119+0000","last_became_peered":"2024-04-26T22:47:41.783119+0000","last_unstale":"2024-04-26T22:47:43.451484+0000","last_undegraded":"2024-04-26T22:47:43.451484+0000","last_fullsized":"2024-04-26T22:47:43.451484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:18.599515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452665+0000","last_change":"2024-04-26T22:47:41.782485+0000","last_active":"2024-04-26T22:47:43.452665+0000","last_peered":"2024-04-26T22:47:43.452665+0000","last_clean":"2024-04-26T22:47:43.452665+0000","last_became_active":"2024-04-26T22:47:41.782118+0000","last_became_peered":"2024-04-26T22:47:41.782118+0000","last_unstale":"2024-04-26T22:47:43.452665+0000","last_undegraded":"2024-04-26T22:47:43.452665+0000","last_fullsized":"2024-04-26T22:47:43.452665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:39.953297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451707+0000","last_change":"2024-04-26T22:47:41.776281+0000","last_active":"2024-04-26T22:47:43.451707+0000","last_peered":"2024-04-26T22:47:43.451707+0000","last_clean":"2024-04-26T22:47:43.451707+0000","last_became_active":"2024-04-26T22:47:41.775838+0000","last_became_peered":"2024-04-26T22:47:41.775838+0000","last_unstale":"2024-04-26T22:47:43.451707+0000","last_undegraded":"2024-04-26T22:47:43.451707+0000","last_fullsized":"2024-04-26T22:47:43.451707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:04.702698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767918+0000","last_change":"2024-04-26T22:47:41.785568+0000","last_active":"2024-04-26T22:47:42.767918+0000","last_peered":"2024-04-26T22:47:42.767918+0000","last_clean":"2024-04-26T22:47:42.767918+0000","last_became_active":"2024-04-26T22:47:41.785188+0000","last_became_peered":"2024-04-26T22:47:41.785188+0000","last_unstale":"2024-04-26T22:47:42.767918+0000","last_undegraded":"2024-04-26T22:47:42.767918+0000","last_fullsized":"2024-04-26T22:47:42.767918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:21.928641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180128+0000","last_change":"2024-04-26T22:47:41.785722+0000","last_active":"2024-04-26T22:47:43.180128+0000","last_peered":"2024-04-26T22:47:43.180128+0000","last_clean":"2024-04-26T22:47:43.180128+0000","last_became_active":"2024-04-26T22:47:41.784935+0000","last_became_peered":"2024-04-26T22:47:41.784935+0000","last_unstale":"2024-04-26T22:47:43.180128+0000","last_undegraded":"2024-04-26T22:47:43.180128+0000","last_fullsized":"2024-04-26T22:47:43.180128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:24.047598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451601+0000","last_change":"2024-04-26T22:47:41.777075+0000","last_active":"2024-04-26T22:47:43.451601+0000","last_peered":"2024-04-26T22:47:43.451601+0000","last_clean":"2024-04-26T22:47:43.451601+0000","last_became_active":"2024-04-26T22:47:41.776734+0000","last_became_peered":"2024-04-26T22:47:41.776734+0000","last_unstale":"2024-04-26T22:47:43.451601+0000","last_undegraded":"2024-04-26T22:47:43.451601+0000","last_fullsized":"2024-04-26T22:47:43.451601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:14.343905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451922+0000","last_change":"2024-04-26T22:47:41.775207+0000","last_active":"2024-04-26T22:47:43.451922+0000","last_peered":"2024-04-26T22:47:43.451922+0000","last_clean":"2024-04-26T22:47:43.451922+0000","last_became_active":"2024-04-26T22:47:41.774834+0000","last_became_peered":"2024-04-26T22:47:41.774834+0000","last_unstale":"2024-04-26T22:47:43.451922+0000","last_undegraded":"2024-04-26T22:47:43.451922+0000","last_fullsized":"2024-04-26T22:47:43.451922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:25.913385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.768518+0000","last_change":"2024-04-26T22:47:41.785894+0000","last_active":"2024-04-26T22:47:42.768518+0000","last_peered":"2024-04-26T22:47:42.768518+0000","last_clean":"2024-04-26T22:47:42.768518+0000","last_became_active":"2024-04-26T22:47:41.785213+0000","last_became_peered":"2024-04-26T22:47:41.785213+0000","last_unstale":"2024-04-26T22:47:42.768518+0000","last_undegraded":"2024-04-26T22:47:42.768518+0000","last_fullsized":"2024-04-26T22:47:42.768518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:44.927776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180604+0000","last_change":"2024-04-26T22:47:41.781532+0000","last_active":"2024-04-26T22:47:43.180604+0000","last_peered":"2024-04-26T22:47:43.180604+0000","last_clean":"2024-04-26T22:47:43.180604+0000","last_became_active":"2024-04-26T22:47:41.781111+0000","last_became_peered":"2024-04-26T22:47:41.781111+0000","last_unstale":"2024-04-26T22:47:43.180604+0000","last_undegraded":"2024-04-26T22:47:43.180604+0000","last_fullsized":"2024-04-26T22:47:43.180604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:54.930233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452656+0000","last_change":"2024-04-26T22:47:41.782289+0000","last_active":"2024-04-26T22:47:43.452656+0000","last_peered":"2024-04-26T22:47:43.452656+0000","last_clean":"2024-04-26T22:47:43.452656+0000","last_became_active":"2024-04-26T22:47:41.781922+0000","last_became_peered":"2024-04-26T22:47:41.781922+0000","last_unstale":"2024-04-26T22:47:43.452656+0000","last_undegraded":"2024-04-26T22:47:43.452656+0000","last_fullsized":"2024-04-26T22:47:43.452656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:30.146485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767338+0000","last_change":"2024-04-26T22:47:41.770970+0000","last_active":"2024-04-26T22:47:42.767338+0000","last_peered":"2024-04-26T22:47:42.767338+0000","last_clean":"2024-04-26T22:47:42.767338+0000","last_became_active":"2024-04-26T22:47:41.770502+0000","last_became_peered":"2024-04-26T22:47:41.770502+0000","last_unstale":"2024-04-26T22:47:42.767338+0000","last_undegraded":"2024-04-26T22:47:42.767338+0000","last_fullsized":"2024-04-26T22:47:42.767338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.988613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451533+0000","last_change":"2024-04-26T22:47:41.779941+0000","last_active":"2024-04-26T22:47:43.451533+0000","last_peered":"2024-04-26T22:47:43.451533+0000","last_clean":"2024-04-26T22:47:43.451533+0000","last_became_active":"2024-04-26T22:47:41.779592+0000","last_became_peered":"2024-04-26T22:47:41.779592+0000","last_unstale":"2024-04-26T22:47:43.451533+0000","last_undegraded":"2024-04-26T22:47:43.451533+0000","last_fullsized":"2024-04-26T22:47:43.451533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:58.628553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451337+0000","last_change":"2024-04-26T22:47:41.781480+0000","last_active":"2024-04-26T22:47:43.451337+0000","last_peered":"2024-04-26T22:47:43.451337+0000","last_clean":"2024-04-26T22:47:43.451337+0000","last_became_active":"2024-04-26T22:47:41.781176+0000","last_became_peered":"2024-04-26T22:47:41.781176+0000","last_unstale":"2024-04-26T22:47:43.451337+0000","last_undegraded":"2024-04-26T22:47:43.451337+0000","last_fullsized":"2024-04-26T22:47:43.451337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:26.361293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451428+0000","last_change":"2024-04-26T22:47:41.771277+0000","last_active":"2024-04-26T22:47:43.451428+0000","last_peered":"2024-04-26T22:47:43.451428+0000","last_clean":"2024-04-26T22:47:43.451428+0000","last_became_active":"2024-04-26T22:47:41.770933+0000","last_became_peered":"2024-04-26T22:47:41.770933+0000","last_unstale":"2024-04-26T22:47:43.451428+0000","last_undegraded":"2024-04-26T22:47:43.451428+0000","last_fullsized":"2024-04-26T22:47:43.451428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:15.045678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.780764+0000","last_change":"2024-04-26T22:47:41.778656+0000","last_active":"2024-04-26T22:47:41.780764+0000","last_peered":"2024-04-26T22:47:41.780764+0000","last_clean":"2024-04-26T22:47:41.780764+0000","last_became_active":"2024-04-26T22:47:41.778282+0000","last_became_peered":"2024-04-26T22:47:41.778282+0000","last_unstale":"2024-04-26T22:47:41.780764+0000","last_undegraded":"2024-04-26T22:47:41.780764+0000","last_fullsized":"2024-04-26T22:47:41.780764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:37.430803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452473+0000","last_change":"2024-04-26T22:47:41.776965+0000","last_active":"2024-04-26T22:47:43.452473+0000","last_peered":"2024-04-26T22:47:43.452473+0000","last_clean":"2024-04-26T22:47:43.452473+0000","last_became_active":"2024-04-26T22:47:41.776625+0000","last_became_peered":"2024-04-26T22:47:41.776625+0000","last_unstale":"2024-04-26T22:47:43.452473+0000","last_undegraded":"2024-04-26T22:47:43.452473+0000","last_fullsized":"2024-04-26T22:47:43.452473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:38.601785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770568+0000","last_change":"2024-04-26T22:47:41.796742+0000","last_active":"2024-04-26T22:47:42.770568+0000","last_peered":"2024-04-26T22:47:42.770568+0000","last_clean":"2024-04-26T22:47:42.770568+0000","last_became_active":"2024-04-26T22:47:41.796362+0000","last_became_peered":"2024-04-26T22:47:41.796362+0000","last_unstale":"2024-04-26T22:47:42.770568+0000","last_undegraded":"2024-04-26T22:47:42.770568+0000","last_fullsized":"2024-04-26T22:47:42.770568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:01:55.092075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.824264+0000","last_change":"2024-04-26T22:47:40.771619+0000","last_active":"2024-04-26T22:47:44.824264+0000","last_peered":"2024-04-26T22:47:44.824264+0000","last_clean":"2024-04-26T22:47:44.824264+0000","last_became_active":"2024-04-26T22:47:40.771262+0000","last_became_peered":"2024-04-26T22:47:40.771262+0000","last_unstale":"2024-04-26T22:47:44.824264+0000","last_undegraded":"2024-04-26T22:47:44.824264+0000","last_fullsized":"2024-04-26T22:47:44.824264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:19.528487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452747+0000","last_change":"2024-04-26T22:47:41.784162+0000","last_active":"2024-04-26T22:47:43.452747+0000","last_peered":"2024-04-26T22:47:43.452747+0000","last_clean":"2024-04-26T22:47:43.452747+0000","last_became_active":"2024-04-26T22:47:41.783821+0000","last_became_peered":"2024-04-26T22:47:41.783821+0000","last_unstale":"2024-04-26T22:47:43.452747+0000","last_undegraded":"2024-04-26T22:47:43.452747+0000","last_fullsized":"2024-04-26T22:47:43.452747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:17:04.237927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452700+0000","last_change":"2024-04-26T22:47:40.779858+0000","last_active":"2024-04-26T22:47:43.452700+0000","last_peered":"2024-04-26T22:47:43.452700+0000","last_clean":"2024-04-26T22:47:43.452700+0000","last_became_active":"2024-04-26T22:47:40.779550+0000","last_became_peered":"2024-04-26T22:47:40.779550+0000","last_unstale":"2024-04-26T22:47:43.452700+0000","last_undegraded":"2024-04-26T22:47:43.452700+0000","last_fullsized":"2024-04-26T22:47:43.452700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:40:54.056426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451574+0000","last_change":"2024-04-26T22:47:41.769865+0000","last_active":"2024-04-26T22:47:43.451574+0000","last_peered":"2024-04-26T22:47:43.451574+0000","last_clean":"2024-04-26T22:47:43.451574+0000","last_became_active":"2024-04-26T22:47:41.769545+0000","last_became_peered":"2024-04-26T22:47:41.769545+0000","last_unstale":"2024-04-26T22:47:43.451574+0000","last_undegraded":"2024-04-26T22:47:43.451574+0000","last_fullsized":"2024-04-26T22:47:43.451574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:39.715241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'11","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.808555+0000","last_change":"2024-04-26T22:47:40.771234+0000","last_active":"2024-04-26T22:47:46.808555+0000","last_peered":"2024-04-26T22:47:46.808555+0000","last_clean":"2024-04-26T22:47:46.808555+0000","last_became_active":"2024-04-26T22:47:40.770876+0000","last_became_peered":"2024-04-26T22:47:40.770876+0000","last_unstale":"2024-04-26T22:47:46.808555+0000","last_undegraded":"2024-04-26T22:47:46.808555+0000","last_fullsized":"2024-04-26T22:47:46.808555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:50:41.272029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451630+0000","last_change":"2024-04-26T22:47:41.770470+0000","last_active":"2024-04-26T22:47:43.451630+0000","last_peered":"2024-04-26T22:47:43.451630+0000","last_clean":"2024-04-26T22:47:43.451630+0000","last_became_active":"2024-04-26T22:47:41.767318+0000","last_became_peered":"2024-04-26T22:47:41.767318+0000","last_unstale":"2024-04-26T22:47:43.451630+0000","last_undegraded":"2024-04-26T22:47:43.451630+0000","last_fullsized":"2024-04-26T22:47:43.451630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:45.324118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.807285+0000","last_change":"2024-04-26T22:47:40.775811+0000","last_active":"2024-04-26T22:47:46.807285+0000","last_peered":"2024-04-26T22:47:46.807285+0000","last_clean":"2024-04-26T22:47:46.807285+0000","last_became_active":"2024-04-26T22:47:40.775349+0000","last_became_peered":"2024-04-26T22:47:40.775349+0000","last_unstale":"2024-04-26T22:47:46.807285+0000","last_undegraded":"2024-04-26T22:47:46.807285+0000","last_fullsized":"2024-04-26T22:47:46.807285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453439+0000","last_change":"2024-04-26T22:47:41.778304+0000","last_active":"2024-04-26T22:47:43.453439+0000","last_peered":"2024-04-26T22:47:43.453439+0000","last_clean":"2024-04-26T22:47:43.453439+0000","last_became_active":"2024-04-26T22:47:41.777930+0000","last_became_peered":"2024-04-26T22:47:41.777930+0000","last_unstale":"2024-04-26T22:47:43.453439+0000","last_undegraded":"2024-04-26T22:47:43.453439+0000","last_fullsized":"2024-04-26T22:47:43.453439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:04.366686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819169+0000","last_change":"2024-04-26T22:47:40.774855+0000","last_active":"2024-04-26T22:47:43.819169+0000","last_peered":"2024-04-26T22:47:43.819169+0000","last_clean":"2024-04-26T22:47:43.819169+0000","last_became_active":"2024-04-26T22:47:40.774517+0000","last_became_peered":"2024-04-26T22:47:40.774517+0000","last_unstale":"2024-04-26T22:47:43.819169+0000","last_undegraded":"2024-04-26T22:47:43.819169+0000","last_fullsized":"2024-04-26T22:47:43.819169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.811072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452357+0000","last_change":"2024-04-26T22:47:41.781432+0000","last_active":"2024-04-26T22:47:43.452357+0000","last_peered":"2024-04-26T22:47:43.452357+0000","last_clean":"2024-04-26T22:47:43.452357+0000","last_became_active":"2024-04-26T22:47:41.781035+0000","last_became_peered":"2024-04-26T22:47:41.781035+0000","last_unstale":"2024-04-26T22:47:43.452357+0000","last_undegraded":"2024-04-26T22:47:43.452357+0000","last_fullsized":"2024-04-26T22:47:43.452357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:31.730087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:45.833589+0000","last_change":"2024-04-26T22:47:40.770419+0000","last_active":"2024-04-26T22:47:45.833589+0000","last_peered":"2024-04-26T22:47:45.833589+0000","last_clean":"2024-04-26T22:47:45.833589+0000","last_became_active":"2024-04-26T22:47:40.769923+0000","last_became_peered":"2024-04-26T22:47:40.769923+0000","last_unstale":"2024-04-26T22:47:45.833589+0000","last_undegraded":"2024-04-26T22:47:45.833589+0000","last_fullsized":"2024-04-26T22:47:45.833589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:15.618904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453022+0000","last_change":"2024-04-26T22:47:41.778334+0000","last_active":"2024-04-26T22:47:43.453022+0000","last_peered":"2024-04-26T22:47:43.453022+0000","last_clean":"2024-04-26T22:47:43.453022+0000","last_became_active":"2024-04-26T22:47:41.777822+0000","last_became_peered":"2024-04-26T22:47:41.777822+0000","last_unstale":"2024-04-26T22:47:43.453022+0000","last_undegraded":"2024-04-26T22:47:43.453022+0000","last_fullsized":"2024-04-26T22:47:43.453022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:57.828165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.823041+0000","last_change":"2024-04-26T22:47:40.779524+0000","last_active":"2024-04-26T22:47:44.823041+0000","last_peered":"2024-04-26T22:47:44.823041+0000","last_clean":"2024-04-26T22:47:44.823041+0000","last_became_active":"2024-04-26T22:47:40.778998+0000","last_became_peered":"2024-04-26T22:47:40.778998+0000","last_unstale":"2024-04-26T22:47:44.823041+0000","last_undegraded":"2024-04-26T22:47:44.823041+0000","last_fullsized":"2024-04-26T22:47:44.823041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:59.979648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451432+0000","last_change":"2024-04-26T22:47:41.768883+0000","last_active":"2024-04-26T22:47:43.451432+0000","last_peered":"2024-04-26T22:47:43.451432+0000","last_clean":"2024-04-26T22:47:43.451432+0000","last_became_active":"2024-04-26T22:47:41.768577+0000","last_became_peered":"2024-04-26T22:47:41.768577+0000","last_unstale":"2024-04-26T22:47:43.451432+0000","last_undegraded":"2024-04-26T22:47:43.451432+0000","last_fullsized":"2024-04-26T22:47:43.451432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:29.267873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452278+0000","last_change":"2024-04-26T22:47:40.774688+0000","last_active":"2024-04-26T22:47:43.452278+0000","last_peered":"2024-04-26T22:47:43.452278+0000","last_clean":"2024-04-26T22:47:43.452278+0000","last_became_active":"2024-04-26T22:47:40.774312+0000","last_became_peered":"2024-04-26T22:47:40.774312+0000","last_unstale":"2024-04-26T22:47:43.452278+0000","last_undegraded":"2024-04-26T22:47:43.452278+0000","last_fullsized":"2024-04-26T22:47:43.452278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:03:34.797965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452470+0000","last_change":"2024-04-26T22:47:41.776948+0000","last_active":"2024-04-26T22:47:43.452470+0000","last_peered":"2024-04-26T22:47:43.452470+0000","last_clean":"2024-04-26T22:47:43.452470+0000","last_became_active":"2024-04-26T22:47:41.776583+0000","last_became_peered":"2024-04-26T22:47:41.776583+0000","last_unstale":"2024-04-26T22:47:43.452470+0000","last_undegraded":"2024-04-26T22:47:43.452470+0000","last_fullsized":"2024-04-26T22:47:43.452470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:42.687249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819273+0000","last_change":"2024-04-26T22:47:40.772060+0000","last_active":"2024-04-26T22:47:43.819273+0000","last_peered":"2024-04-26T22:47:43.819273+0000","last_clean":"2024-04-26T22:47:43.819273+0000","last_became_active":"2024-04-26T22:47:40.771690+0000","last_became_peered":"2024-04-26T22:47:40.771690+0000","last_unstale":"2024-04-26T22:47:43.819273+0000","last_undegraded":"2024-04-26T22:47:43.819273+0000","last_fullsized":"2024-04-26T22:47:43.819273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:53.619399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451834+0000","last_change":"2024-04-26T22:47:41.776902+0000","last_active":"2024-04-26T22:47:43.451834+0000","last_peered":"2024-04-26T22:47:43.451834+0000","last_clean":"2024-04-26T22:47:43.451834+0000","last_became_active":"2024-04-26T22:47:41.776475+0000","last_became_peered":"2024-04-26T22:47:41.776475+0000","last_unstale":"2024-04-26T22:47:43.451834+0000","last_undegraded":"2024-04-26T22:47:43.451834+0000","last_fullsized":"2024-04-26T22:47:43.451834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:57:17.674715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.810754+0000","last_change":"2024-04-26T22:47:40.775828+0000","last_active":"2024-04-26T22:47:46.810754+0000","last_peered":"2024-04-26T22:47:46.810754+0000","last_clean":"2024-04-26T22:47:46.810754+0000","last_became_active":"2024-04-26T22:47:40.775439+0000","last_became_peered":"2024-04-26T22:47:40.775439+0000","last_unstale":"2024-04-26T22:47:46.810754+0000","last_undegraded":"2024-04-26T22:47:46.810754+0000","last_fullsized":"2024-04-26T22:47:46.810754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:28:21.061843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451452+0000","last_change":"2024-04-26T22:47:41.782731+0000","last_active":"2024-04-26T22:47:43.451452+0000","last_peered":"2024-04-26T22:47:43.451452+0000","last_clean":"2024-04-26T22:47:43.451452+0000","last_became_active":"2024-04-26T22:47:41.781390+0000","last_became_peered":"2024-04-26T22:47:41.781390+0000","last_unstale":"2024-04-26T22:47:43.451452+0000","last_undegraded":"2024-04-26T22:47:43.451452+0000","last_fullsized":"2024-04-26T22:47:43.451452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:15.349252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453381+0000","last_change":"2024-04-26T22:47:40.770488+0000","last_active":"2024-04-26T22:47:43.453381+0000","last_peered":"2024-04-26T22:47:43.453381+0000","last_clean":"2024-04-26T22:47:43.453381+0000","last_became_active":"2024-04-26T22:47:40.770049+0000","last_became_peered":"2024-04-26T22:47:40.770049+0000","last_unstale":"2024-04-26T22:47:43.453381+0000","last_undegraded":"2024-04-26T22:47:43.453381+0000","last_fullsized":"2024-04-26T22:47:43.453381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:03.391917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451889+0000","last_change":"2024-04-26T22:47:41.778433+0000","last_active":"2024-04-26T22:47:43.451889+0000","last_peered":"2024-04-26T22:47:43.451889+0000","last_clean":"2024-04-26T22:47:43.451889+0000","last_became_active":"2024-04-26T22:47:41.777603+0000","last_became_peered":"2024-04-26T22:47:41.777603+0000","last_unstale":"2024-04-26T22:47:43.451889+0000","last_undegraded":"2024-04-26T22:47:43.451889+0000","last_fullsized":"2024-04-26T22:47:43.451889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:02.206291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.816917+0000","last_change":"2024-04-26T22:47:40.774417+0000","last_active":"2024-04-26T22:47:43.816917+0000","last_peered":"2024-04-26T22:47:43.816917+0000","last_clean":"2024-04-26T22:47:43.816917+0000","last_became_active":"2024-04-26T22:47:40.773932+0000","last_became_peered":"2024-04-26T22:47:40.773932+0000","last_unstale":"2024-04-26T22:47:43.816917+0000","last_undegraded":"2024-04-26T22:47:43.816917+0000","last_fullsized":"2024-04-26T22:47:43.816917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:15.694401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452087+0000","last_change":"2024-04-26T22:47:41.787466+0000","last_active":"2024-04-26T22:47:43.452087+0000","last_peered":"2024-04-26T22:47:43.452087+0000","last_clean":"2024-04-26T22:47:43.452087+0000","last_became_active":"2024-04-26T22:47:41.787105+0000","last_became_peered":"2024-04-26T22:47:41.787105+0000","last_unstale":"2024-04-26T22:47:43.452087+0000","last_undegraded":"2024-04-26T22:47:43.452087+0000","last_fullsized":"2024-04-26T22:47:43.452087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:48:34.656495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.823363+0000","last_change":"2024-04-26T22:47:40.767810+0000","last_active":"2024-04-26T22:47:44.823363+0000","last_peered":"2024-04-26T22:47:44.823363+0000","last_clean":"2024-04-26T22:47:44.823363+0000","last_became_active":"2024-04-26T22:47:40.767508+0000","last_became_peered":"2024-04-26T22:47:40.767508+0000","last_unstale":"2024-04-26T22:47:44.823363+0000","last_undegraded":"2024-04-26T22:47:44.823363+0000","last_fullsized":"2024-04-26T22:47:44.823363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:32.023939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767072+0000","last_change":"2024-04-26T22:47:41.774134+0000","last_active":"2024-04-26T22:47:42.767072+0000","last_peered":"2024-04-26T22:47:42.767072+0000","last_clean":"2024-04-26T22:47:42.767072+0000","last_became_active":"2024-04-26T22:47:41.773613+0000","last_became_peered":"2024-04-26T22:47:41.773613+0000","last_unstale":"2024-04-26T22:47:42.767072+0000","last_undegraded":"2024-04-26T22:47:42.767072+0000","last_fullsized":"2024-04-26T22:47:42.767072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:41:51.317633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.817168+0000","last_change":"2024-04-26T22:47:40.776836+0000","last_active":"2024-04-26T22:47:43.817168+0000","last_peered":"2024-04-26T22:47:43.817168+0000","last_clean":"2024-04-26T22:47:43.817168+0000","last_became_active":"2024-04-26T22:47:40.776493+0000","last_became_peered":"2024-04-26T22:47:40.776493+0000","last_unstale":"2024-04-26T22:47:43.817168+0000","last_undegraded":"2024-04-26T22:47:43.817168+0000","last_fullsized":"2024-04-26T22:47:43.817168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:55.695456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180520+0000","last_change":"2024-04-26T22:47:41.784562+0000","last_active":"2024-04-26T22:47:43.180520+0000","last_peered":"2024-04-26T22:47:43.180520+0000","last_clean":"2024-04-26T22:47:43.180520+0000","last_became_active":"2024-04-26T22:47:41.784150+0000","last_became_peered":"2024-04-26T22:47:41.784150+0000","last_unstale":"2024-04-26T22:47:43.180520+0000","last_undegraded":"2024-04-26T22:47:43.180520+0000","last_fullsized":"2024-04-26T22:47:43.180520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:35.247848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452024+0000","last_change":"2024-04-26T22:47:40.768984+0000","last_active":"2024-04-26T22:47:43.452024+0000","last_peered":"2024-04-26T22:47:43.452024+0000","last_clean":"2024-04-26T22:47:43.452024+0000","last_became_active":"2024-04-26T22:47:40.768596+0000","last_became_peered":"2024-04-26T22:47:40.768596+0000","last_unstale":"2024-04-26T22:47:43.452024+0000","last_undegraded":"2024-04-26T22:47:43.452024+0000","last_fullsized":"2024-04-26T22:47:43.452024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:44.099790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180027+0000","last_change":"2024-04-26T22:47:41.778100+0000","last_active":"2024-04-26T22:47:43.180027+0000","last_peered":"2024-04-26T22:47:43.180027+0000","last_clean":"2024-04-26T22:47:43.180027+0000","last_became_active":"2024-04-26T22:47:41.777775+0000","last_became_peered":"2024-04-26T22:47:41.777775+0000","last_unstale":"2024-04-26T22:47:43.180027+0000","last_undegraded":"2024-04-26T22:47:43.180027+0000","last_fullsized":"2024-04-26T22:47:43.180027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:52.698089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.798848+0000","last_change":"2024-04-26T22:47:40.761904+0000","last_active":"2024-04-26T22:47:46.798848+0000","last_peered":"2024-04-26T22:47:46.798848+0000","last_clean":"2024-04-26T22:47:46.798848+0000","last_became_active":"2024-04-26T22:47:40.761591+0000","last_became_peered":"2024-04-26T22:47:40.761591+0000","last_unstale":"2024-04-26T22:47:46.798848+0000","last_undegraded":"2024-04-26T22:47:46.798848+0000","last_fullsized":"2024-04-26T22:47:46.798848+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:55.774669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452047+0000","last_change":"2024-04-26T22:47:41.780703+0000","last_active":"2024-04-26T22:47:43.452047+0000","last_peered":"2024-04-26T22:47:43.452047+0000","last_clean":"2024-04-26T22:47:43.452047+0000","last_became_active":"2024-04-26T22:47:41.775023+0000","last_became_peered":"2024-04-26T22:47:41.775023+0000","last_unstale":"2024-04-26T22:47:43.452047+0000","last_undegraded":"2024-04-26T22:47:43.452047+0000","last_fullsized":"2024-04-26T22:47:43.452047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:09.515514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.799020+0000","last_change":"2024-04-26T22:47:40.766095+0000","last_active":"2024-04-26T22:47:46.799020+0000","last_peered":"2024-04-26T22:47:46.799020+0000","last_clean":"2024-04-26T22:47:46.799020+0000","last_became_active":"2024-04-26T22:47:40.765751+0000","last_became_peered":"2024-04-26T22:47:40.765751+0000","last_unstale":"2024-04-26T22:47:46.799020+0000","last_undegraded":"2024-04-26T22:47:46.799020+0000","last_fullsized":"2024-04-26T22:47:46.799020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:22.877861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.786084+0000","last_change":"2024-04-26T22:47:41.781793+0000","last_active":"2024-04-26T22:47:41.786084+0000","last_peered":"2024-04-26T22:47:41.786084+0000","last_clean":"2024-04-26T22:47:41.786084+0000","last_became_active":"2024-04-26T22:47:41.781470+0000","last_became_peered":"2024-04-26T22:47:41.781470+0000","last_unstale":"2024-04-26T22:47:41.786084+0000","last_undegraded":"2024-04-26T22:47:41.786084+0000","last_fullsized":"2024-04-26T22:47:41.786084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:00.668938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452183+0000","last_change":"2024-04-26T22:47:40.767364+0000","last_active":"2024-04-26T22:47:43.452183+0000","last_peered":"2024-04-26T22:47:43.452183+0000","last_clean":"2024-04-26T22:47:43.452183+0000","last_became_active":"2024-04-26T22:47:40.766907+0000","last_became_peered":"2024-04-26T22:47:40.766907+0000","last_unstale":"2024-04-26T22:47:43.452183+0000","last_undegraded":"2024-04-26T22:47:43.452183+0000","last_fullsized":"2024-04-26T22:47:43.452183+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:56.408996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.768356+0000","last_change":"2024-04-26T22:47:41.784095+0000","last_active":"2024-04-26T22:47:42.768356+0000","last_peered":"2024-04-26T22:47:42.768356+0000","last_clean":"2024-04-26T22:47:42.768356+0000","last_became_active":"2024-04-26T22:47:41.783749+0000","last_became_peered":"2024-04-26T22:47:41.783749+0000","last_unstale":"2024-04-26T22:47:42.768356+0000","last_undegraded":"2024-04-26T22:47:42.768356+0000","last_fullsized":"2024-04-26T22:47:42.768356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:11:26.530223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.811482+0000","last_change":"2024-04-26T22:47:40.778988+0000","last_active":"2024-04-26T22:47:46.811482+0000","last_peered":"2024-04-26T22:47:46.811482+0000","last_clean":"2024-04-26T22:47:46.811482+0000","last_became_active":"2024-04-26T22:47:40.778686+0000","last_became_peered":"2024-04-26T22:47:40.778686+0000","last_unstale":"2024-04-26T22:47:46.811482+0000","last_undegraded":"2024-04-26T22:47:46.811482+0000","last_fullsized":"2024-04-26T22:47:46.811482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.525866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767942+0000","last_change":"2024-04-26T22:47:41.787168+0000","last_active":"2024-04-26T22:47:42.767942+0000","last_peered":"2024-04-26T22:47:42.767942+0000","last_clean":"2024-04-26T22:47:42.767942+0000","last_became_active":"2024-04-26T22:47:41.786811+0000","last_became_peered":"2024-04-26T22:47:41.786811+0000","last_unstale":"2024-04-26T22:47:42.767942+0000","last_undegraded":"2024-04-26T22:47:42.767942+0000","last_fullsized":"2024-04-26T22:47:42.767942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:34.578058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452435+0000","last_change":"2024-04-26T22:47:41.780945+0000","last_active":"2024-04-26T22:47:43.452435+0000","last_peered":"2024-04-26T22:47:43.452435+0000","last_clean":"2024-04-26T22:47:43.452435+0000","last_became_active":"2024-04-26T22:47:41.779325+0000","last_became_peered":"2024-04-26T22:47:41.779325+0000","last_unstale":"2024-04-26T22:47:43.452435+0000","last_undegraded":"2024-04-26T22:47:43.452435+0000","last_fullsized":"2024-04-26T22:47:43.452435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:57.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.824536+0000","last_change":"2024-04-26T22:47:40.781706+0000","last_active":"2024-04-26T22:47:44.824536+0000","last_peered":"2024-04-26T22:47:44.824536+0000","last_clean":"2024-04-26T22:47:44.824536+0000","last_became_active":"2024-04-26T22:47:40.781014+0000","last_became_peered":"2024-04-26T22:47:40.781014+0000","last_unstale":"2024-04-26T22:47:44.824536+0000","last_undegraded":"2024-04-26T22:47:44.824536+0000","last_fullsized":"2024-04-26T22:47:44.824536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:17:16.523767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452727+0000","last_change":"2024-04-26T22:47:41.772852+0000","last_active":"2024-04-26T22:47:43.452727+0000","last_peered":"2024-04-26T22:47:43.452727+0000","last_clean":"2024-04-26T22:47:43.452727+0000","last_became_active":"2024-04-26T22:47:41.772502+0000","last_became_peered":"2024-04-26T22:47:41.772502+0000","last_unstale":"2024-04-26T22:47:43.452727+0000","last_undegraded":"2024-04-26T22:47:43.452727+0000","last_fullsized":"2024-04-26T22:47:43.452727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:36.870593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819431+0000","last_change":"2024-04-26T22:47:40.768151+0000","last_active":"2024-04-26T22:47:43.819431+0000","last_peered":"2024-04-26T22:47:43.819431+0000","last_clean":"2024-04-26T22:47:43.819431+0000","last_became_active":"2024-04-26T22:47:40.767756+0000","last_became_peered":"2024-04-26T22:47:40.767756+0000","last_unstale":"2024-04-26T22:47:43.819431+0000","last_undegraded":"2024-04-26T22:47:43.819431+0000","last_fullsized":"2024-04-26T22:47:43.819431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:38:09.059145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":337,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180017+0000","last_change":"2024-04-26T22:46:24.327741+0000","last_active":"2024-04-26T22:47:43.180017+0000","last_peered":"2024-04-26T22:47:43.180017+0000","last_clean":"2024-04-26T22:47:43.180017+0000","last_became_active":"2024-04-26T22:46:24.327280+0000","last_became_peered":"2024-04-26T22:46:24.327280+0000","last_unstale":"2024-04-26T22:47:43.180017+0000","last_undegraded":"2024-04-26T22:47:43.180017+0000","last_fullsized":"2024-04-26T22:47:43.180017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_clean_scrub_stamp":"2024-04-26T22:42:55.984240+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:47.094131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.766860+0000","last_change":"2024-04-26T22:47:41.779195+0000","last_active":"2024-04-26T22:47:42.766860+0000","last_peered":"2024-04-26T22:47:42.766860+0000","last_clean":"2024-04-26T22:47:42.766860+0000","last_became_active":"2024-04-26T22:47:41.778682+0000","last_became_peered":"2024-04-26T22:47:41.778682+0000","last_unstale":"2024-04-26T22:47:42.766860+0000","last_undegraded":"2024-04-26T22:47:42.766860+0000","last_fullsized":"2024-04-26T22:47:42.766860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:37.983340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.821930+0000","last_change":"2024-04-26T22:47:40.774817+0000","last_active":"2024-04-26T22:47:44.821930+0000","last_peered":"2024-04-26T22:47:44.821930+0000","last_clean":"2024-04-26T22:47:44.821930+0000","last_became_active":"2024-04-26T22:47:40.772660+0000","last_became_peered":"2024-04-26T22:47:40.772660+0000","last_unstale":"2024-04-26T22:47:44.821930+0000","last_undegraded":"2024-04-26T22:47:44.821930+0000","last_fullsized":"2024-04-26T22:47:44.821930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:30.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:46.795377+0000","last_change":"2024-04-26T22:47:40.773595+0000","last_active":"2024-04-26T22:47:46.795377+0000","last_peered":"2024-04-26T22:47:46.795377+0000","last_clean":"2024-04-26T22:47:46.795377+0000","last_became_active":"2024-04-26T22:47:40.773199+0000","last_became_peered":"2024-04-26T22:47:40.773199+0000","last_unstale":"2024-04-26T22:47:46.795377+0000","last_undegraded":"2024-04-26T22:47:46.795377+0000","last_fullsized":"2024-04-26T22:47:46.795377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:19:31.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767273+0000","last_change":"2024-04-26T22:47:41.780428+0000","last_active":"2024-04-26T22:47:42.767273+0000","last_peered":"2024-04-26T22:47:42.767273+0000","last_clean":"2024-04-26T22:47:42.767273+0000","last_became_active":"2024-04-26T22:47:41.779468+0000","last_became_peered":"2024-04-26T22:47:41.779468+0000","last_unstale":"2024-04-26T22:47:42.767273+0000","last_undegraded":"2024-04-26T22:47:42.767273+0000","last_fullsized":"2024-04-26T22:47:42.767273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:05.067347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.823351+0000","last_change":"2024-04-26T22:47:40.770624+0000","last_active":"2024-04-26T22:47:44.823351+0000","last_peered":"2024-04-26T22:47:44.823351+0000","last_clean":"2024-04-26T22:47:44.823351+0000","last_became_active":"2024-04-26T22:47:40.770250+0000","last_became_peered":"2024-04-26T22:47:40.770250+0000","last_unstale":"2024-04-26T22:47:44.823351+0000","last_undegraded":"2024-04-26T22:47:44.823351+0000","last_fullsized":"2024-04-26T22:47:44.823351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:16:53.527984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452706+0000","last_change":"2024-04-26T22:47:41.773667+0000","last_active":"2024-04-26T22:47:43.452706+0000","last_peered":"2024-04-26T22:47:43.452706+0000","last_clean":"2024-04-26T22:47:43.452706+0000","last_became_active":"2024-04-26T22:47:41.772876+0000","last_became_peered":"2024-04-26T22:47:41.772876+0000","last_unstale":"2024-04-26T22:47:43.452706+0000","last_undegraded":"2024-04-26T22:47:43.452706+0000","last_fullsized":"2024-04-26T22:47:43.452706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.363992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.818808+0000","last_change":"2024-04-26T22:47:40.774090+0000","last_active":"2024-04-26T22:47:43.818808+0000","last_peered":"2024-04-26T22:47:43.818808+0000","last_clean":"2024-04-26T22:47:43.818808+0000","last_became_active":"2024-04-26T22:47:40.773543+0000","last_became_peered":"2024-04-26T22:47:40.773543+0000","last_unstale":"2024-04-26T22:47:43.818808+0000","last_undegraded":"2024-04-26T22:47:43.818808+0000","last_fullsized":"2024-04-26T22:47:43.818808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:17:06.907699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452376+0000","last_change":"2024-04-26T22:47:41.781231+0000","last_active":"2024-04-26T22:47:43.452376+0000","last_peered":"2024-04-26T22:47:43.452376+0000","last_clean":"2024-04-26T22:47:43.452376+0000","last_became_active":"2024-04-26T22:47:41.780837+0000","last_became_peered":"2024-04-26T22:47:41.780837+0000","last_unstale":"2024-04-26T22:47:43.452376+0000","last_undegraded":"2024-04-26T22:47:43.452376+0000","last_fullsized":"2024-04-26T22:47:43.452376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:24.491628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.818199+0000","last_change":"2024-04-26T22:47:40.771454+0000","last_active":"2024-04-26T22:47:43.818199+0000","last_peered":"2024-04-26T22:47:43.818199+0000","last_clean":"2024-04-26T22:47:43.818199+0000","last_became_active":"2024-04-26T22:47:40.770488+0000","last_became_peered":"2024-04-26T22:47:40.770488+0000","last_unstale":"2024-04-26T22:47:43.818199+0000","last_undegraded":"2024-04-26T22:47:43.818199+0000","last_fullsized":"2024-04-26T22:47:43.818199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:19.422331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451669+0000","last_change":"2024-04-26T22:47:41.776683+0000","last_active":"2024-04-26T22:47:43.451669+0000","last_peered":"2024-04-26T22:47:43.451669+0000","last_clean":"2024-04-26T22:47:43.451669+0000","last_became_active":"2024-04-26T22:47:41.776309+0000","last_became_peered":"2024-04-26T22:47:41.776309+0000","last_unstale":"2024-04-26T22:47:43.451669+0000","last_undegraded":"2024-04-26T22:47:43.451669+0000","last_fullsized":"2024-04-26T22:47:43.451669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:45.604270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.818910+0000","last_change":"2024-04-26T22:47:40.774392+0000","last_active":"2024-04-26T22:47:43.818910+0000","last_peered":"2024-04-26T22:47:43.818910+0000","last_clean":"2024-04-26T22:47:43.818910+0000","last_became_active":"2024-04-26T22:47:40.773682+0000","last_became_peered":"2024-04-26T22:47:40.773682+0000","last_unstale":"2024-04-26T22:47:43.818910+0000","last_undegraded":"2024-04-26T22:47:43.818910+0000","last_fullsized":"2024-04-26T22:47:43.818910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:06:56.769791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.785944+0000","last_change":"2024-04-26T22:47:41.784191+0000","last_active":"2024-04-26T22:47:41.785944+0000","last_peered":"2024-04-26T22:47:41.785944+0000","last_clean":"2024-04-26T22:47:41.785944+0000","last_became_active":"2024-04-26T22:47:41.783884+0000","last_became_peered":"2024-04-26T22:47:41.783884+0000","last_unstale":"2024-04-26T22:47:41.785944+0000","last_undegraded":"2024-04-26T22:47:41.785944+0000","last_fullsized":"2024-04-26T22:47:41.785944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:50:41.479348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770146+0000","last_change":"2024-04-26T22:47:40.767796+0000","last_active":"2024-04-26T22:47:42.770146+0000","last_peered":"2024-04-26T22:47:42.770146+0000","last_clean":"2024-04-26T22:47:42.770146+0000","last_became_active":"2024-04-26T22:47:40.767450+0000","last_became_peered":"2024-04-26T22:47:40.767450+0000","last_unstale":"2024-04-26T22:47:42.770146+0000","last_undegraded":"2024-04-26T22:47:42.770146+0000","last_fullsized":"2024-04-26T22:47:42.770146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:41.481981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452200+0000","last_change":"2024-04-26T22:47:41.774308+0000","last_active":"2024-04-26T22:47:43.452200+0000","last_peered":"2024-04-26T22:47:43.452200+0000","last_clean":"2024-04-26T22:47:43.452200+0000","last_became_active":"2024-04-26T22:47:41.773713+0000","last_became_peered":"2024-04-26T22:47:41.773713+0000","last_unstale":"2024-04-26T22:47:43.452200+0000","last_undegraded":"2024-04-26T22:47:43.452200+0000","last_fullsized":"2024-04-26T22:47:43.452200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:11.186258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.819268+0000","last_change":"2024-04-26T22:47:40.768761+0000","last_active":"2024-04-26T22:47:43.819268+0000","last_peered":"2024-04-26T22:47:43.819268+0000","last_clean":"2024-04-26T22:47:43.819268+0000","last_became_active":"2024-04-26T22:47:40.768434+0000","last_became_peered":"2024-04-26T22:47:40.768434+0000","last_unstale":"2024-04-26T22:47:43.819268+0000","last_undegraded":"2024-04-26T22:47:43.819268+0000","last_fullsized":"2024-04-26T22:47:43.819268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:01.631990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180613+0000","last_change":"2024-04-26T22:47:41.778214+0000","last_active":"2024-04-26T22:47:43.180613+0000","last_peered":"2024-04-26T22:47:43.180613+0000","last_clean":"2024-04-26T22:47:43.180613+0000","last_became_active":"2024-04-26T22:47:41.775889+0000","last_became_peered":"2024-04-26T22:47:41.775889+0000","last_unstale":"2024-04-26T22:47:43.180613+0000","last_undegraded":"2024-04-26T22:47:43.180613+0000","last_fullsized":"2024-04-26T22:47:43.180613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:24.654113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:44.822543+0000","last_change":"2024-04-26T22:47:40.772366+0000","last_active":"2024-04-26T22:47:44.822543+0000","last_peered":"2024-04-26T22:47:44.822543+0000","last_clean":"2024-04-26T22:47:44.822543+0000","last_became_active":"2024-04-26T22:47:40.772035+0000","last_became_peered":"2024-04-26T22:47:40.772035+0000","last_unstale":"2024-04-26T22:47:44.822543+0000","last_undegraded":"2024-04-26T22:47:44.822543+0000","last_fullsized":"2024-04-26T22:47:44.822543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:54.429132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180656+0000","last_change":"2024-04-26T22:47:41.777074+0000","last_active":"2024-04-26T22:47:43.180656+0000","last_peered":"2024-04-26T22:47:43.180656+0000","last_clean":"2024-04-26T22:47:43.180656+0000","last_became_active":"2024-04-26T22:47:41.776225+0000","last_became_peered":"2024-04-26T22:47:41.776225+0000","last_unstale":"2024-04-26T22:47:43.180656+0000","last_undegraded":"2024-04-26T22:47:43.180656+0000","last_fullsized":"2024-04-26T22:47:43.180656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:06:59.801103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.453137+0000","last_change":"2024-04-26T22:47:40.776174+0000","last_active":"2024-04-26T22:47:43.453137+0000","last_peered":"2024-04-26T22:47:43.453137+0000","last_clean":"2024-04-26T22:47:43.453137+0000","last_became_active":"2024-04-26T22:47:40.775866+0000","last_became_peered":"2024-04-26T22:47:40.775866+0000","last_unstale":"2024-04-26T22:47:43.453137+0000","last_undegraded":"2024-04-26T22:47:43.453137+0000","last_fullsized":"2024-04-26T22:47:43.453137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:01.306261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180284+0000","last_change":"2024-04-26T22:47:41.777225+0000","last_active":"2024-04-26T22:47:43.180284+0000","last_peered":"2024-04-26T22:47:43.180284+0000","last_clean":"2024-04-26T22:47:43.180284+0000","last_became_active":"2024-04-26T22:47:41.776814+0000","last_became_peered":"2024-04-26T22:47:41.776814+0000","last_unstale":"2024-04-26T22:47:43.180284+0000","last_undegraded":"2024-04-26T22:47:43.180284+0000","last_fullsized":"2024-04-26T22:47:43.180284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:30.254893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767205+0000","last_change":"2024-04-26T22:47:40.781678+0000","last_active":"2024-04-26T22:47:42.767205+0000","last_peered":"2024-04-26T22:47:42.767205+0000","last_clean":"2024-04-26T22:47:42.767205+0000","last_became_active":"2024-04-26T22:47:40.781156+0000","last_became_peered":"2024-04-26T22:47:40.781156+0000","last_unstale":"2024-04-26T22:47:42.767205+0000","last_undegraded":"2024-04-26T22:47:42.767205+0000","last_fullsized":"2024-04-26T22:47:42.767205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:53.353768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.180523+0000","last_change":"2024-04-26T22:47:41.780116+0000","last_active":"2024-04-26T22:47:43.180523+0000","last_peered":"2024-04-26T22:47:43.180523+0000","last_clean":"2024-04-26T22:47:43.180523+0000","last_became_active":"2024-04-26T22:47:41.777443+0000","last_became_peered":"2024-04-26T22:47:41.777443+0000","last_unstale":"2024-04-26T22:47:43.180523+0000","last_undegraded":"2024-04-26T22:47:43.180523+0000","last_fullsized":"2024-04-26T22:47:43.180523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:22.948100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.785849+0000","last_change":"2024-04-26T22:47:41.784618+0000","last_active":"2024-04-26T22:47:41.785849+0000","last_peered":"2024-04-26T22:47:41.785849+0000","last_clean":"2024-04-26T22:47:41.785849+0000","last_became_active":"2024-04-26T22:47:41.784269+0000","last_became_peered":"2024-04-26T22:47:41.784269+0000","last_unstale":"2024-04-26T22:47:41.785849+0000","last_undegraded":"2024-04-26T22:47:41.785849+0000","last_fullsized":"2024-04-26T22:47:41.785849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:33:19.337891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770269+0000","last_change":"2024-04-26T22:47:41.786063+0000","last_active":"2024-04-26T22:47:42.770269+0000","last_peered":"2024-04-26T22:47:42.770269+0000","last_clean":"2024-04-26T22:47:42.770269+0000","last_became_active":"2024-04-26T22:47:41.785657+0000","last_became_peered":"2024-04-26T22:47:41.785657+0000","last_unstale":"2024-04-26T22:47:42.770269+0000","last_undegraded":"2024-04-26T22:47:42.770269+0000","last_fullsized":"2024-04-26T22:47:42.770269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:14.289477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770170+0000","last_change":"2024-04-26T22:47:41.788788+0000","last_active":"2024-04-26T22:47:42.770170+0000","last_peered":"2024-04-26T22:47:42.770170+0000","last_clean":"2024-04-26T22:47:42.770170+0000","last_became_active":"2024-04-26T22:47:41.788434+0000","last_became_peered":"2024-04-26T22:47:41.788434+0000","last_unstale":"2024-04-26T22:47:42.770170+0000","last_undegraded":"2024-04-26T22:47:42.770170+0000","last_fullsized":"2024-04-26T22:47:42.770170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:05.056380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.451742+0000","last_change":"2024-04-26T22:47:41.780435+0000","last_active":"2024-04-26T22:47:43.451742+0000","last_peered":"2024-04-26T22:47:43.451742+0000","last_clean":"2024-04-26T22:47:43.451742+0000","last_became_active":"2024-04-26T22:47:41.780126+0000","last_became_peered":"2024-04-26T22:47:41.780126+0000","last_unstale":"2024-04-26T22:47:43.451742+0000","last_undegraded":"2024-04-26T22:47:43.451742+0000","last_fullsized":"2024-04-26T22:47:43.451742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:20.061814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767447+0000","last_change":"2024-04-26T22:47:41.775797+0000","last_active":"2024-04-26T22:47:42.767447+0000","last_peered":"2024-04-26T22:47:42.767447+0000","last_clean":"2024-04-26T22:47:42.767447+0000","last_became_active":"2024-04-26T22:47:41.775286+0000","last_became_peered":"2024-04-26T22:47:41.775286+0000","last_unstale":"2024-04-26T22:47:42.767447+0000","last_undegraded":"2024-04-26T22:47:42.767447+0000","last_fullsized":"2024-04-26T22:47:42.767447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:45.290373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.786030+0000","last_change":"2024-04-26T22:47:41.784714+0000","last_active":"2024-04-26T22:47:41.786030+0000","last_peered":"2024-04-26T22:47:41.786030+0000","last_clean":"2024-04-26T22:47:41.786030+0000","last_became_active":"2024-04-26T22:47:41.783233+0000","last_became_peered":"2024-04-26T22:47:41.783233+0000","last_unstale":"2024-04-26T22:47:41.786030+0000","last_undegraded":"2024-04-26T22:47:41.786030+0000","last_fullsized":"2024-04-26T22:47:41.786030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:24.132003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.770703+0000","last_change":"2024-04-26T22:47:41.796781+0000","last_active":"2024-04-26T22:47:42.770703+0000","last_peered":"2024-04-26T22:47:42.770703+0000","last_clean":"2024-04-26T22:47:42.770703+0000","last_became_active":"2024-04-26T22:47:41.796326+0000","last_became_peered":"2024-04-26T22:47:41.796326+0000","last_unstale":"2024-04-26T22:47:42.770703+0000","last_undegraded":"2024-04-26T22:47:42.770703+0000","last_fullsized":"2024-04-26T22:47:42.770703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:30:59.924808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452602+0000","last_change":"2024-04-26T22:47:41.785149+0000","last_active":"2024-04-26T22:47:43.452602+0000","last_peered":"2024-04-26T22:47:43.452602+0000","last_clean":"2024-04-26T22:47:43.452602+0000","last_became_active":"2024-04-26T22:47:41.784807+0000","last_became_peered":"2024-04-26T22:47:41.784807+0000","last_unstale":"2024-04-26T22:47:43.452602+0000","last_undegraded":"2024-04-26T22:47:43.452602+0000","last_fullsized":"2024-04-26T22:47:43.452602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:45.521731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452445+0000","last_change":"2024-04-26T22:47:41.768789+0000","last_active":"2024-04-26T22:47:43.452445+0000","last_peered":"2024-04-26T22:47:43.452445+0000","last_clean":"2024-04-26T22:47:43.452445+0000","last_became_active":"2024-04-26T22:47:41.768466+0000","last_became_peered":"2024-04-26T22:47:41.768466+0000","last_unstale":"2024-04-26T22:47:43.452445+0000","last_undegraded":"2024-04-26T22:47:43.452445+0000","last_fullsized":"2024-04-26T22:47:43.452445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:15.798301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452319+0000","last_change":"2024-04-26T22:47:41.781867+0000","last_active":"2024-04-26T22:47:43.452319+0000","last_peered":"2024-04-26T22:47:43.452319+0000","last_clean":"2024-04-26T22:47:43.452319+0000","last_became_active":"2024-04-26T22:47:41.780648+0000","last_became_peered":"2024-04-26T22:47:41.780648+0000","last_unstale":"2024-04-26T22:47:43.452319+0000","last_undegraded":"2024-04-26T22:47:43.452319+0000","last_fullsized":"2024-04-26T22:47:43.452319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:25.393892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:43.452129+0000","last_change":"2024-04-26T22:47:41.774622+0000","last_active":"2024-04-26T22:47:43.452129+0000","last_peered":"2024-04-26T22:47:43.452129+0000","last_clean":"2024-04-26T22:47:43.452129+0000","last_became_active":"2024-04-26T22:47:41.774262+0000","last_became_peered":"2024-04-26T22:47:41.774262+0000","last_unstale":"2024-04-26T22:47:43.452129+0000","last_undegraded":"2024-04-26T22:47:43.452129+0000","last_fullsized":"2024-04-26T22:47:43.452129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:32.571497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T22:47:42.767433+0000","last_change":"2024-04-26T22:47:41.768981+0000","last_active":"2024-04-26T22:47:42.767433+0000","last_peered":"2024-04-26T22:47:42.767433+0000","last_clean":"2024-04-26T22:47:42.767433+0000","last_became_active":"2024-04-26T22:47:41.768515+0000","last_became_peered":"2024-04-26T22:47:41.768515+0000","last_unstale":"2024-04-26T22:47:42.767433+0000","last_undegraded":"2024-04-26T22:47:42.767433+0000","last_fullsized":"2024-04-26T22:47:42.767433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:43:47.680248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-26T22:47:41.789688+0000","last_change":"2024-04-26T22:47:41.788415+0000","last_active":"2024-04-26T22:47:41.789688+0000","last_peered":"2024-04-26T22:47:41.789688+0000","last_clean":"2024-04-26T22:47:41.789688+0000","last_became_active":"2024-04-26T22:47:41.788076+0000","last_became_peered":"2024-04-26T22:47:41.788076+0000","last_unstale":"2024-04-26T22:47:41.789688+0000","last_undegraded":"2024-04-26T22:47:41.789688+0000","last_fullsized":"2024-04-26T22:47:41.789688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:20.829264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":288005,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5950,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":17603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1032203,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":439305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1032422,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":438633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":440715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":439215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":440789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":3314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T22:47:49.645 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T22:47:49.645 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T22:47:49.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T22:47:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:47:50.051 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:49 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3409494320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:50.051 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:49 smithi001 ceph-mon[29435]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:47:50.051 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:49 smithi001 ceph-mon[29435]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:47:50.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:49 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3409494320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:50.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:49 smithi038 ceph-mon[32433]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:47:50.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:49 smithi038 ceph-mon[32433]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:47:50.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:49 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3409494320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:50.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:49 smithi186 ceph-mon[32774]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:47:50.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:49 smithi186 ceph-mon[32774]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:47:51.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:50 smithi001 ceph-mon[29435]: pgmap v295: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 5.6 KiB/s wr, 24 op/s 2024-04-26T22:47:51.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:50 smithi038 ceph-mon[32433]: pgmap v295: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 5.6 KiB/s wr, 24 op/s 2024-04-26T22:47:51.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:50 smithi186 ceph-mon[32774]: pgmap v295: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 5.6 KiB/s wr, 24 op/s 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 -- 172.21.15.1:0/395623432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238106860 msgr2=0x7f8238108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 --2- 172.21.15.1:0/395623432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238106860 0x7f8238108c50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8220009990 tx=0x7f8220009ca0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 -- 172.21.15.1:0/395623432 shutdown_connections 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 --2- 172.21.15.1:0/395623432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238103f30 0x7f8238106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 --2- 172.21.15.1:0/395623432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8238109190 0x7f823810b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 --2- 172.21.15.1:0/395623432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238106860 0x7f8238108c50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 -- 172.21.15.1:0/395623432 >> 172.21.15.1:0/395623432 conn(0x7f82380fd940 msgr2=0x7f82380ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 -- 172.21.15.1:0/395623432 shutdown_connections 2024-04-26T22:47:51.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.672+0000 7f823dc90700 1 -- 172.21.15.1:0/395623432 wait complete. 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 Processor -- start 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 -- start start 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8238103f30 0x7f82381994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238106860 0x7f8238199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 0x7f82381a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f823810ef60 con 0x7f8238109190 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f823810ede0 con 0x7f8238103f30 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f823dc90700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f823810f0e0 con 0x7f8238106860 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f8237fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 0x7f82381a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f8237fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 0x7f82381a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36876/0 (socket says 172.21.15.1:36876) 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.673+0000 7f8237fff700 1 -- 172.21.15.1:0/4106408918 learned_addr learned my addr 172.21.15.1:0/4106408918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8236ffd700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238106860 0x7f8238199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:51.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f82377fe700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8238103f30 0x7f82381994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8237fff700 1 -- 172.21.15.1:0/4106408918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238106860 msgr2=0x7f8238199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8237fff700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238106860 0x7f8238199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8237fff700 1 -- 172.21.15.1:0/4106408918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8238103f30 msgr2=0x7f82381994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8237fff700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8238103f30 0x7f82381994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8237fff700 1 -- 172.21.15.1:0/4106408918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82381a1130 con 0x7f8238109190 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8236ffd700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238106860 0x7f8238199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8237fff700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 0x7f82381a0a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f822800c900 tx=0x7f822800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228004460 con 0x7f8238109190 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8228003b30 con 0x7f8238109190 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82381a1420 con 0x7f8238109190 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.674+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822800ec50 con 0x7f8238109190 2024-04-26T22:47:51.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.675+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82381a1910 con 0x7f8238109190 2024-04-26T22:47:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.676+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8238051960 con 0x7f8238109190 2024-04-26T22:47:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.677+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8228003680 con 0x7f8238109190 2024-04-26T22:47:51.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.677+0000 7f8234ff9700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82240a3e00 0x7f82240a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.678+0000 7f82377fe700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82240a3e00 0x7f82240a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.678+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f82280d0170 con 0x7f8238109190 2024-04-26T22:47:51.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.678+0000 7f82377fe700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82240a3e00 0x7f82240a62c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8230009550 tx=0x7f82300091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:51.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.682+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8228013030 con 0x7f8238109190 2024-04-26T22:47:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.978+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f823802e4c0 con 0x7f8238109190 2024-04-26T22:47:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.981+0000 7f8234ff9700 1 -- 172.21.15.1:0/4106408918 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f822809cba0 con 0x7f8238109190 2024-04-26T22:47:51.981 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-04-26T22:47:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAXLyxmPzRsOhAA8ec0F+mwne4VlypbiBa3VQ== 2024-04-26T22:47:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82240a3e00 msgr2=0x7f82240a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82240a3e00 0x7f82240a62c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8230009550 tx=0x7f82300091b0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 msgr2=0x7f82381a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 0x7f82381a0a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f822800c900 tx=0x7f822800cc10 comp rx=0 tx=0).stop 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 shutdown_connections 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82240a3e00 0x7f82240a62c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238109190 0x7f82381a0a90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8238103f30 0x7f82381994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 --2- 172.21.15.1:0/4106408918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8238106860 0x7f8238199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 >> 172.21.15.1:0/4106408918 conn(0x7f82380fd940 msgr2=0x7f8238104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 shutdown_connections 2024-04-26T22:47:51.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:51.983+0000 7f823dc90700 1 -- 172.21.15.1:0/4106408918 wait complete. 2024-04-26T22:47:52.775 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:47:52.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T22:47:52.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T22:47:52.814 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T22:47:52.814 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T22:47:52.814 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T22:47:52.839 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 22:47:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T22:47:52.556+0000 7f79095f7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-26T22:47:53.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:52 smithi001 ceph-mon[29435]: pgmap v296: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-26T22:47:53.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:52 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4106408918' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T22:47:53.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:52 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4106408918' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T22:47:53.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:52 smithi038 ceph-mon[32433]: pgmap v296: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-26T22:47:53.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:52 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/4106408918' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T22:47:53.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:52 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/4106408918' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T22:47:53.102 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:47:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:52 smithi186 ceph-mon[32774]: pgmap v296: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-26T22:47:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:52 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/4106408918' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T22:47:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:52 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/4106408918' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T22:47:54.877 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:54 smithi001 ceph-mon[29435]: pgmap v297: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.6 KiB/s wr, 25 op/s 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 -- 172.21.15.1:0/3253183444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c108740 msgr2=0x7f325c110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 --2- 172.21.15.1:0/3253183444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c108740 0x7f325c110350 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f325400b0d0 tx=0x7f325400b3e0 comp rx=0 tx=0).stop 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 -- 172.21.15.1:0/3253183444 shutdown_connections 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 --2- 172.21.15.1:0/3253183444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c06bdf0 0x7f325c06c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 --2- 172.21.15.1:0/3253183444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c108740 0x7f325c110350 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 --2- 172.21.15.1:0/3253183444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f325c06c7a0 0x7f325c108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 -- 172.21.15.1:0/3253183444 >> 172.21.15.1:0/3253183444 conn(0x7f325c078370 msgr2=0x7f325c078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 -- 172.21.15.1:0/3253183444 shutdown_connections 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.983+0000 7f326430e700 1 -- 172.21.15.1:0/3253183444 wait complete. 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 Processor -- start 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 -- start start 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f325c06bdf0 0x7f325c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 0x7f325c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c108740 0x7f325c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f325c113690 con 0x7f325c06c7a0 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f325c113510 con 0x7f325c06bdf0 2024-04-26T22:47:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f326430e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f325c113810 con 0x7f325c108740 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f32618a9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 0x7f325c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f32618a9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 0x7f325c19fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37132/0 (socket says 172.21.15.1:37132) 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f32618a9700 1 -- 172.21.15.1:0/2258099012 learned_addr learned my addr 172.21.15.1:0/2258099012 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f32620aa700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f325c06bdf0 0x7f325c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.984+0000 7f32628ab700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c108740 0x7f325c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32618a9700 1 -- 172.21.15.1:0/2258099012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c108740 msgr2=0x7f325c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32618a9700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c108740 0x7f325c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32618a9700 1 -- 172.21.15.1:0/2258099012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f325c06bdf0 msgr2=0x7f325c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32618a9700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f325c06bdf0 0x7f325c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32618a9700 1 -- 172.21.15.1:0/2258099012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c19a0d0 con 0x7f325c06c7a0 2024-04-26T22:47:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32618a9700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 0x7f325c19fca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f324c00d980 tx=0x7f324c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c017070 con 0x7f325c06c7a0 2024-04-26T22:47:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f324c009de0 con 0x7f325c06c7a0 2024-04-26T22:47:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325c19a3c0 con 0x7f325c06c7a0 2024-04-26T22:47:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c00f3c0 con 0x7f325c06c7a0 2024-04-26T22:47:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.985+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f325c1a6630 con 0x7f325c06c7a0 2024-04-26T22:47:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.986+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325c06a110 con 0x7f325c06c7a0 2024-04-26T22:47:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.987+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f324c009930 con 0x7f325c06c7a0 2024-04-26T22:47:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.988+0000 7f32537fe700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f32480a3d50 0x7f32480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.988+0000 7f32620aa700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f32480a3d50 0x7f32480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.988+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f324c0cfe40 con 0x7f325c06c7a0 2024-04-26T22:47:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.988+0000 7f32620aa700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f32480a3d50 0x7f32480a6210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f325c078b70 tx=0x7f3258009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:54.992+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f324c09c870 con 0x7f325c06c7a0 2024-04-26T22:47:55.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:54 smithi038 ceph-mon[32433]: pgmap v297: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.6 KiB/s wr, 25 op/s 2024-04-26T22:47:55.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:54 smithi186 ceph-mon[32774]: pgmap v297: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.6 KiB/s wr, 25 op/s 2024-04-26T22:47:55.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.259+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f325c19ae50 con 0x7f325c06c7a0 2024-04-26T22:47:55.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.266+0000 7f32537fe700 1 -- 172.21.15.1:0/2258099012 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264987 (secure 0 0 0) 0x7f324c09c170 con 0x7f325c06c7a0 2024-04-26T22:47:55.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f32480a3d50 msgr2=0x7f32480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f32480a3d50 0x7f32480a6210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f325c078b70 tx=0x7f3258009190 comp rx=0 tx=0).stop 2024-04-26T22:47:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 msgr2=0x7f325c19fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 0x7f325c19fca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f324c00d980 tx=0x7f324c00dc90 comp rx=0 tx=0).stop 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 shutdown_connections 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f32480a3d50 0x7f32480a6210 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c108740 0x7f325c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f325c06bdf0 0x7f325c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 --2- 172.21.15.1:0/2258099012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f325c06c7a0 0x7f325c19fca0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.268+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 >> 172.21.15.1:0/2258099012 conn(0x7f325c078370 msgr2=0x7f325c100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.269+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 shutdown_connections 2024-04-26T22:47:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:55.269+0000 7f326430e700 1 -- 172.21.15.1:0/2258099012 wait complete. 2024-04-26T22:47:56.113 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:55 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2258099012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T22:47:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":4125558078},{"type":"v1","addr":"172.21.15.1:6801","nonce":4125558078}]},"active_addr":"172.21.15.1:6801/4125558078","active_change":"2024-04-26T22:40:29.666311+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.1:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3636295822}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":539625582}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":154950408}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2762207166}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3248225075}]}]} 2024-04-26T22:47:56.121 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T22:47:56.121 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T22:47:56.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T22:47:56.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:55 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2258099012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T22:47:56.285 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:55 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2258099012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T22:47:56.373 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:47:57.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:56 smithi001 ceph-mon[29435]: pgmap v298: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.8 KiB/s wr, 25 op/s 2024-04-26T22:47:57.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:56 smithi186 ceph-mon[32774]: pgmap v298: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.8 KiB/s wr, 25 op/s 2024-04-26T22:47:57.259 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:56 smithi038 ceph-mon[32433]: pgmap v298: 97 pgs: 97 active+clean; 621 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.8 KiB/s wr, 25 op/s 2024-04-26T22:47:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 -- 172.21.15.1:0/4290023499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc108500 msgr2=0x7f55bc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/4290023499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc108500 0x7f55bc1088e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009990 tx=0x7f55ac009ca0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 -- 172.21.15.1:0/4290023499 shutdown_connections 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/4290023499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55bc1027a0 0x7f55bc110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/4290023499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc108500 0x7f55bc1088e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/4290023499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc101de0 0x7f55bc102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 -- 172.21.15.1:0/4290023499 >> 172.21.15.1:0/4290023499 conn(0x7f55bc077640 msgr2=0x7f55bc077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 -- 172.21.15.1:0/4290023499 shutdown_connections 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 -- 172.21.15.1:0/4290023499 wait complete. 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.159+0000 7f55c1ef9700 1 Processor -- start 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 -- start start 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 0x7f55bc10f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc1027a0 0x7f55bc10fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55bc108500 0x7f55bc1a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc113640 con 0x7f55bc101de0 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1134c0 con 0x7f55bc1027a0 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55c1ef9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1137c0 con 0x7f55bc108500 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 0x7f55bc10f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 0x7f55bc10f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37144/0 (socket says 172.21.15.1:37144) 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 -- 172.21.15.1:0/351623645 learned_addr learned my addr 172.21.15.1:0/351623645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55baffd700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc1027a0 0x7f55bc10fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bbfff700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55bc108500 0x7f55bc1a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 -- 172.21.15.1:0/351623645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55bc108500 msgr2=0x7f55bc1a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55bc108500 0x7f55bc1a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 -- 172.21.15.1:0/351623645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc1027a0 msgr2=0x7f55bc10fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc1027a0 0x7f55bc10fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 -- 172.21.15.1:0/351623645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc1a5100 con 0x7f55bc101de0 2024-04-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.160+0000 7f55bb7fe700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 0x7f55bc10f550 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00b5c0 tx=0x7f55ac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.161+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac003680 con 0x7f55bc101de0 2024-04-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.161+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc1a5390 con 0x7f55bc101de0 2024-04-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.161+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55ac004ce0 con 0x7f55bc101de0 2024-04-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.161+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac003cd0 con 0x7f55bc101de0 2024-04-26T22:47:58.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.161+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc1acca0 con 0x7f55bc101de0 2024-04-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.162+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55bc06b610 con 0x7f55bc101de0 2024-04-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.163+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f55ac004830 con 0x7f55bc101de0 2024-04-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.163+0000 7f55b8ff9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f55a40a3e70 0x7f55a40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.163+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f55ac0d11e0 con 0x7f55bc101de0 2024-04-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.163+0000 7f55baffd700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f55a40a3e70 0x7f55a40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:47:58.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.164+0000 7f55baffd700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f55a40a3e70 0x7f55a40a6330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f55bc104f50 tx=0x7f55b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:47:58.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.168+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55ac013070 con 0x7f55bc101de0 2024-04-26T22:47:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.376+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55bc1ace70 con 0x7f55bc101de0 2024-04-26T22:47:58.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.378+0000 7f55b8ff9700 1 -- 172.21.15.1:0/351623645 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22040 (secure 0 0 0) 0x7f55ac09dca0 con 0x7f55bc101de0 2024-04-26T22:47:58.378 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:47:58.378 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":85,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:47:42.763750+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:47:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f55a40a3e70 msgr2=0x7f55a40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f55a40a3e70 0x7f55a40a6330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f55bc104f50 tx=0x7f55b000b410 comp rx=0 tx=0).stop 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 msgr2=0x7f55bc10f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 0x7f55bc10f550 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00b5c0 tx=0x7f55ac0047b0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 shutdown_connections 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f55a40a3e70 0x7f55a40a6330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55bc108500 0x7f55bc1a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc101de0 0x7f55bc10f550 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 --2- 172.21.15.1:0/351623645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc1027a0 0x7f55bc10fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 >> 172.21.15.1:0/351623645 conn(0x7f55bc077640 msgr2=0x7f55bc100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 shutdown_connections 2024-04-26T22:47:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:47:58.380+0000 7f55c1ef9700 1 -- 172.21.15.1:0/351623645 wait complete. 2024-04-26T22:47:59.031 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T22:47:59.031 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T22:47:59.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:58 smithi186 ceph-mon[32774]: pgmap v299: 97 pgs: 97 active+clean; 627 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 23 op/s 2024-04-26T22:47:59.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:47:58 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/351623645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:59.281 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:47:59.345 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:58 smithi001 ceph-mon[29435]: pgmap v299: 97 pgs: 97 active+clean; 627 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 23 op/s 2024-04-26T22:47:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:47:58 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/351623645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:47:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:58 smithi038 ceph-mon[32433]: pgmap v299: 97 pgs: 97 active+clean; 627 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 23 op/s 2024-04-26T22:47:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:47:58 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/351623645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:48:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.110+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/3540199813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48105e60 msgr2=0x7f3f481062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.110+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/3540199813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48105e60 0x7f3f481062f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000b0a0 tx=0x7f3f4000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/3540199813 shutdown_connections 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/3540199813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f48108350 0x7f3f4806b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/3540199813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48105e60 0x7f3f481062f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/3540199813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f4806b670 0x7f3f4806bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/3540199813 >> 172.21.15.1:0/3540199813 conn(0x7f3f480792d0 msgr2=0x7f3f480796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/3540199813 shutdown_connections 2024-04-26T22:48:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.111+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/3540199813 wait complete. 2024-04-26T22:48:01.112 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:00 smithi001 ceph-mon[29435]: pgmap v300: 97 pgs: 97 active+clean; 627 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 Processor -- start 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 -- start start 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4806b670 0x7f3f481a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 0x7f3f481a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48108350 0x7f3f481a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48119c10 con 0x7f3f48108350 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48119a90 con 0x7f3f48105e60 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4c9f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48119d90 con 0x7f3f4806b670 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f45d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 0x7f3f481a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f46d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48108350 0x7f3f481a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f45d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 0x7f3f481a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50944/0 (socket says 172.21.15.1:50944) 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f45d9b700 1 -- 172.21.15.1:0/2201176835 learned_addr learned my addr 172.21.15.1:0/2201176835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.112+0000 7f3f4659c700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4806b670 0x7f3f481a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f45d9b700 1 -- 172.21.15.1:0/2201176835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4806b670 msgr2=0x7f3f481a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f45d9b700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4806b670 0x7f3f481a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f45d9b700 1 -- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48108350 msgr2=0x7f3f481a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f45d9b700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48108350 0x7f3f481a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f45d9b700 1 -- 172.21.15.1:0/2201176835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f481a5090 con 0x7f3f48105e60 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f46d9d700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48108350 0x7f3f481a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f45d9b700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 0x7f3f481a05c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00d950 tx=0x7f3f3c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c016070 con 0x7f3f48105e60 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f481a5380 con 0x7f3f48105e60 2024-04-26T22:48:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f3c00fd40 con 0x7f3f48105e60 2024-04-26T22:48:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c00e3b0 con 0x7f3f48105e60 2024-04-26T22:48:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.113+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f481acc00 con 0x7f3f48105e60 2024-04-26T22:48:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.114+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4806a110 con 0x7f3f48105e60 2024-04-26T22:48:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.115+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3f3c019070 con 0x7f3f48105e60 2024-04-26T22:48:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.116+0000 7f3f377fe700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3f300a3cd0 0x7f3f300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.116+0000 7f3f4659c700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3f300a3cd0 0x7f3f300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.116+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3f3c0cff80 con 0x7f3f48105e60 2024-04-26T22:48:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.117+0000 7f3f4659c700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3f300a3cd0 0x7f3f300a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009a10 tx=0x7f3f38009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.121+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3f3c09c8b0 con 0x7f3f48105e60 2024-04-26T22:48:01.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:00 smithi186 ceph-mon[32774]: pgmap v300: 97 pgs: 97 active+clean; 627 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-04-26T22:48:01.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.329+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f481ace00 con 0x7f3f48105e60 2024-04-26T22:48:01.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.331+0000 7f3f377fe700 1 -- 172.21.15.1:0/2201176835 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22040 (secure 0 0 0) 0x7f3f3c09c1b0 con 0x7f3f48105e60 2024-04-26T22:48:01.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:48:01.332 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":85,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:47:42.763750+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3f300a3cd0 msgr2=0x7f3f300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3f300a3cd0 0x7f3f300a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009a10 tx=0x7f3f38009190 comp rx=0 tx=0).stop 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 msgr2=0x7f3f481a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 0x7f3f481a05c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00d950 tx=0x7f3f3c00dc60 comp rx=0 tx=0).stop 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 shutdown_connections 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3f300a3cd0 0x7f3f300a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4806b670 0x7f3f481a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f48108350 0x7f3f481a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 --2- 172.21.15.1:0/2201176835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f48105e60 0x7f3f481a05c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 >> 172.21.15.1:0/2201176835 conn(0x7f3f480792d0 msgr2=0x7f3f48104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 shutdown_connections 2024-04-26T22:48:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:01.333+0000 7f3f4c9f1700 1 -- 172.21.15.1:0/2201176835 wait complete. 2024-04-26T22:48:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:00 smithi038 ceph-mon[32433]: pgmap v300: 97 pgs: 97 active+clean; 627 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-04-26T22:48:02.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T22:48:02.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T22:48:02.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T22:48:02.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T22:48:02.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T22:48:02.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T22:48:02.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T22:48:02.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T22:48:02.047 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T22:48:02.047 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T22:48:02.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T22:48:02.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T22:48:02.135 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:01 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2201176835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:48:02.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:01 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2201176835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:48:02.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:01 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2201176835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:48:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:02.852 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:02.852 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:02.852 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:03.149 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:02 smithi001 ceph-mon[29435]: pgmap v301: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 KiB/s wr, 17 op/s 2024-04-26T22:48:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:02 smithi186 ceph-mon[32774]: pgmap v301: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 KiB/s wr, 17 op/s 2024-04-26T22:48:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:02 smithi038 ceph-mon[32433]: pgmap v301: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 KiB/s wr, 17 op/s 2024-04-26T22:48:03.481 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:03.482 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:03.482 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:03.484 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:03.488 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:03.664 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:05.165 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:04 smithi001 ceph-mon[29435]: pgmap v302: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-26T22:48:05.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:04 smithi186 ceph-mon[32774]: pgmap v302: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-26T22:48:05.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:04 smithi038 ceph-mon[32433]: pgmap v302: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-26T22:48:07.034 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:06 smithi001 ceph-mon[29435]: pgmap v303: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-26T22:48:07.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:06 smithi038 ceph-mon[32433]: pgmap v303: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-26T22:48:07.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:06 smithi186 ceph-mon[32774]: pgmap v303: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-26T22:48:09.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:08 smithi001 ceph-mon[29435]: pgmap v304: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-26T22:48:09.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:08 smithi186 ceph-mon[32774]: pgmap v304: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-26T22:48:09.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:08 smithi038 ceph-mon[32433]: pgmap v304: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-26T22:48:10.778 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:11.046 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:10 smithi001 ceph-mon[29435]: pgmap v305: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:48:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:10 smithi186 ceph-mon[32774]: pgmap v305: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:48:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:10 smithi038 ceph-mon[32433]: pgmap v305: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:48:11.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 -- 172.21.15.1:0/1333603133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0108390 msgr2=0x7fe7d0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1333603133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0108390 0x7fe7d0108770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009990 tx=0x7fe7c0009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 -- 172.21.15.1:0/1333603133 shutdown_connections 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1333603133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7d0102ce0 0x7fe7d010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1333603133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0108390 0x7fe7d0108770 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1333603133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7d0102320 0x7fe7d01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 -- 172.21.15.1:0/1333603133 >> 172.21.15.1:0/1333603133 conn(0x7fe7d00792d0 msgr2=0x7fe7d00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 -- 172.21.15.1:0/1333603133 shutdown_connections 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.470+0000 7fe7d5134700 1 -- 172.21.15.1:0/1333603133 wait complete. 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 Processor -- start 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 -- start start 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7d0102320 0x7fe7d019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:11.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0102ce0 0x7fe7d019fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7d0108390 0x7fe7d0199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d0112580 con 0x7fe7d0102ce0 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d0112400 con 0x7fe7d0102320 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7d5134700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d0112700 con 0x7fe7d0108390 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7ce59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0102ce0 0x7fe7d019fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7ce59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0102ce0 0x7fe7d019fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57794/0 (socket says 172.21.15.1:57794) 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7ce59c700 1 -- 172.21.15.1:0/1410233005 learned_addr learned my addr 172.21.15.1:0/1410233005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7ced9d700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7d0102320 0x7fe7d019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.471+0000 7fe7cf59e700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7d0108390 0x7fe7d0199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7ce59c700 1 -- 172.21.15.1:0/1410233005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7d0108390 msgr2=0x7fe7d0199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7ce59c700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7d0108390 0x7fe7d0199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7ce59c700 1 -- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7d0102320 msgr2=0x7fe7d019f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7ce59c700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7d0102320 0x7fe7d019f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7ce59c700 1 -- 172.21.15.1:0/1410233005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d019a170 con 0x7fe7d0102ce0 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7ce59c700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0102ce0 0x7fe7d019fd90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400b530 tx=0x7fe7c400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4015020 con 0x7fe7d0102ce0 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7c4011ce0 con 0x7fe7d0102ce0 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d019a460 con 0x7fe7d0102ce0 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4010dd0 con 0x7fe7d0102ce0 2024-04-26T22:48:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.472+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d01a6720 con 0x7fe7d0102ce0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.473+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe7d0107460 con 0x7fe7d0102ce0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.474+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fe7c4018020 con 0x7fe7d0102ce0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7b7fff700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe7b80a3cd0 0x7fe7b80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7ced9d700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe7b80a3cd0 0x7fe7b80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe7c40d0140 con 0x7fe7d0102ce0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7b7fff700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] conn(0x7fe7b80ad4b0 0x7fe7b80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 --> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7c401ce70 con 0x7fe7b80ad4b0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fe7c40d0620 con 0x7fe7d0102ce0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7ced9d700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe7b80a3cd0 0x7fe7b80a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009750 tx=0x7fe7c0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.475+0000 7fe7cf59e700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] conn(0x7fe7b80ad4b0 0x7fe7b80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.476+0000 7fe7cf59e700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] conn(0x7fe7b80ad4b0 0x7fe7b80af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:11.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.477+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== osd.10 v2:172.21.15.38:6824/1475724574 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7c401ce70 con 0x7fe7b80ad4b0 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.493+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 --> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe7d0103a40 con 0x7fe7b80ad4b0 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.493+0000 7fe7b7fff700 1 -- 172.21.15.1:0/1410233005 <== osd.10 v2:172.21.15.38:6824/1475724574 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe7d0103a40 con 0x7fe7b80ad4b0 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] conn(0x7fe7b80ad4b0 msgr2=0x7fe7b80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] conn(0x7fe7b80ad4b0 0x7fe7b80af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe7b80a3cd0 msgr2=0x7fe7b80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe7b80a3cd0 0x7fe7b80a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009750 tx=0x7fe7c0005fd0 comp rx=0 tx=0).stop 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0102ce0 msgr2=0x7fe7d019fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 --2- 172.21.15.1:0/1410233005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7d0102ce0 0x7fe7d019fd90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400b530 tx=0x7fe7c400b840 comp rx=0 tx=0).stop 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7cf59e700 1 -- 172.21.15.1:0/1410233005 reap_dead start 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 shutdown_connections 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 >> 172.21.15.1:0/1410233005 conn(0x7fe7d00792d0 msgr2=0x7fe7d006b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 shutdown_connections 2024-04-26T22:48:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:11.494+0000 7fe7d5134700 1 -- 172.21.15.1:0/1410233005 wait complete. 2024-04-26T22:48:12.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 -- 172.21.15.1:0/2406162950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 msgr2=0x7fec481102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/2406162950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec481102c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fec4000b0a0 tx=0x7fec4000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 -- 172.21.15.1:0/2406162950 shutdown_connections 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/2406162950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec481102c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/2406162950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec48109480 0x7fec48109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/2406162950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec481034e0 0x7fec48103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 -- 172.21.15.1:0/2406162950 >> 172.21.15.1:0/2406162950 conn(0x7fec4807a5e0 msgr2=0x7fec4807a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 -- 172.21.15.1:0/2406162950 shutdown_connections 2024-04-26T22:48:12.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.056+0000 7fec4ef8b700 1 -- 172.21.15.1:0/2406162950 wait complete. 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 Processor -- start 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 -- start start 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec481034e0 0x7fec48075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec48075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48109480 0x7fec480761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fec48113690 con 0x7fec48103ea0 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fec48113510 con 0x7fec481034e0 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4ef8b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec48113810 con 0x7fec48109480 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec47fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec48075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec47fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec48075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34782/0 (socket says 172.21.15.1:34782) 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec47fff700 1 -- 172.21.15.1:0/1381920547 learned_addr learned my addr 172.21.15.1:0/1381920547 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.057+0000 7fec4cd27700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec481034e0 0x7fec48075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:12.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec4d528700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48109480 0x7fec480761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec47fff700 1 -- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48109480 msgr2=0x7fec480761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec47fff700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48109480 0x7fec480761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec47fff700 1 -- 172.21.15.1:0/1381920547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec481034e0 msgr2=0x7fec48075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec47fff700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec481034e0 0x7fec48075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec47fff700 1 -- 172.21.15.1:0/1381920547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec481a8120 con 0x7fec48103ea0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec4d528700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48109480 0x7fec480761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec47fff700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec48075c60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b530 tx=0x7fec3c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c015020 con 0x7fec48103ea0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec481a8410 con 0x7fec48103ea0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec3c011ce0 con 0x7fec48103ea0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c010e40 con 0x7fec48103ea0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.058+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec481a8950 con 0x7fec48103ea0 2024-04-26T22:48:12.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.059+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec48063b30 con 0x7fec48103ea0 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.060+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fec3c018020 con 0x7fec48103ea0 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec45ffb700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fec300a3cd0 0x7fec300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec4cd27700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fec300a3cd0 0x7fec300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fec3c0d0200 con 0x7fec48103ea0 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec45ffb700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] conn(0x7fec300ad4b0 0x7fec300af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 --> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec300aff80 con 0x7fec300ad4b0 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fec3c0d06e0 con 0x7fec48103ea0 2024-04-26T22:48:12.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.061+0000 7fec4cd27700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fec300a3cd0 0x7fec300a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fec38009a10 tx=0x7fec38009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:12.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.062+0000 7fec4d528700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] conn(0x7fec300ad4b0 0x7fec300af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:12.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.062+0000 7fec4d528700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] conn(0x7fec300ad4b0 0x7fec300af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:12.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.063+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== osd.2 v2:172.21.15.186:6800/2937417141 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec300aff80 con 0x7fec300ad4b0 2024-04-26T22:48:12.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.079+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 --> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec48063f30 con 0x7fec300ad4b0 2024-04-26T22:48:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec45ffb700 1 -- 172.21.15.1:0/1381920547 <== osd.2 v2:172.21.15.186:6800/2937417141 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fec48063f30 con 0x7fec300ad4b0 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] conn(0x7fec300ad4b0 msgr2=0x7fec300af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] conn(0x7fec300ad4b0 0x7fec300af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fec300a3cd0 msgr2=0x7fec300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fec300a3cd0 0x7fec300a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fec38009a10 tx=0x7fec38009190 comp rx=0 tx=0).stop 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 msgr2=0x7fec48075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 --2- 172.21.15.1:0/1381920547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec48103ea0 0x7fec48075c60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b530 tx=0x7fec3c00b840 comp rx=0 tx=0).stop 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4d528700 1 -- 172.21.15.1:0/1381920547 reap_dead start 2024-04-26T22:48:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 shutdown_connections 2024-04-26T22:48:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 >> 172.21.15.1:0/1381920547 conn(0x7fec4807a5e0 msgr2=0x7fec481015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 shutdown_connections 2024-04-26T22:48:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:12.080+0000 7fec4ef8b700 1 -- 172.21.15.1:0/1381920547 wait complete. 2024-04-26T22:48:13.150 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:12 smithi001 ceph-mon[29435]: pgmap v306: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-26T22:48:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:12 smithi186 ceph-mon[32774]: pgmap v306: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-26T22:48:13.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:12 smithi038 ceph-mon[32433]: pgmap v306: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-26T22:48:15.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:14 smithi001 ceph-mon[29435]: pgmap v307: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:15.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:14 smithi186 ceph-mon[32774]: pgmap v307: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:14 smithi038 ceph-mon[32433]: pgmap v307: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:16.220 INFO:teuthology.orchestra.run.smithi001.stdout:77309411395 2024-04-26T22:48:16.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T22:48:17.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:16 smithi001 ceph-mon[29435]: pgmap v308: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.122+0000 7f8bacee0700 1 -- 172.21.15.1:0/3320628228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba81073a0 msgr2=0x7f8ba8107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.122+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3320628228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba81073a0 0x7f8ba8107780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8b980099c0 tx=0x7f8b98009cd0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.122+0000 7f8bacee0700 1 -- 172.21.15.1:0/3320628228 shutdown_connections 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.122+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3320628228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ba8103030 0x7f8ba810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3320628228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba81073a0 0x7f8ba8107780 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3320628228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ba8102670 0x7f8ba8102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 -- 172.21.15.1:0/3320628228 >> 172.21.15.1:0/3320628228 conn(0x7f8ba8078190 msgr2=0x7f8ba80785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 -- 172.21.15.1:0/3320628228 shutdown_connections 2024-04-26T22:48:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 -- 172.21.15.1:0/3320628228 wait complete. 2024-04-26T22:48:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 Processor -- start 2024-04-26T22:48:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.123+0000 7f8bacee0700 1 -- start start 2024-04-26T22:48:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ba8102670 0x7f8ba8104050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba8103030 0x7f8ba8104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ba81073a0 0x7f8ba8104ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba8112550 con 0x7f8ba8103030 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba81123d0 con 0x7f8ba8102670 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba81126d0 con 0x7f8ba81073a0 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba8103030 0x7f8ba8104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba8103030 0x7f8ba8104590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34800/0 (socket says 172.21.15.1:34800) 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 -- 172.21.15.1:0/3905467456 learned_addr learned my addr 172.21.15.1:0/3905467456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8ba659c700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ba8102670 0x7f8ba8104050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8ba6d9d700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ba81073a0 0x7f8ba8104ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 -- 172.21.15.1:0/3905467456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ba81073a0 msgr2=0x7f8ba8104ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ba81073a0 0x7f8ba8104ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 -- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ba8102670 msgr2=0x7f8ba8104050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ba8102670 0x7f8ba8104050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 -- 172.21.15.1:0/3905467456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba81a8050 con 0x7f8ba8103030 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9ffff700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba8103030 0x7f8ba8104590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000d950 tx=0x7f8b9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90016070 con 0x7f8ba8103030 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba81a8340 con 0x7f8ba8103030 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b9000fd40 con 0x7f8ba8103030 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.124+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000e3b0 con 0x7f8ba8103030 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.125+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba81a88d0 con 0x7f8ba8103030 2024-04-26T22:48:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.125+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ba81007b0 con 0x7f8ba8103030 2024-04-26T22:48:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8b90019070 con 0x7f8ba8103030 2024-04-26T22:48:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8b9f7fe700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b940a3cd0 0x7f8b940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8b900d0380 con 0x7f8ba8103030 2024-04-26T22:48:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8ba659c700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b940a3cd0 0x7f8b940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8b9f7fe700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] conn(0x7f8b940ad4b0 0x7f8b940af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 --> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b90013ec0 con 0x7f8b940ad4b0 2024-04-26T22:48:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.127+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f8b900d0860 con 0x7f8ba8103030 2024-04-26T22:48:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.128+0000 7f8ba659c700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b940a3cd0 0x7f8b940a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009990 tx=0x7f8b98009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.128+0000 7f8ba6d9d700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] conn(0x7f8b940ad4b0 0x7f8b940af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.128+0000 7f8ba6d9d700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] conn(0x7f8b940ad4b0 0x7f8b940af8d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.129+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== osd.4 v2:172.21.15.38:6808/414739057 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8b90013ec0 con 0x7f8b940ad4b0 2024-04-26T22:48:17.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.146+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 --> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8ba8103a90 con 0x7f8b940ad4b0 2024-04-26T22:48:17.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.147+0000 7f8b9f7fe700 1 -- 172.21.15.1:0/3905467456 <== osd.4 v2:172.21.15.38:6808/414739057 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8ba8103a90 con 0x7f8b940ad4b0 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.147+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] conn(0x7f8b940ad4b0 msgr2=0x7f8b940af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.147+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] conn(0x7f8b940ad4b0 0x7f8b940af8d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b940a3cd0 msgr2=0x7f8b940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8b940a3cd0 0x7f8b940a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009990 tx=0x7f8b98009650 comp rx=0 tx=0).stop 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba8103030 msgr2=0x7f8ba8104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 --2- 172.21.15.1:0/3905467456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ba8103030 0x7f8ba8104590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000d950 tx=0x7f8b9000dc60 comp rx=0 tx=0).stop 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8ba6d9d700 1 -- 172.21.15.1:0/3905467456 reap_dead start 2024-04-26T22:48:17.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 shutdown_connections 2024-04-26T22:48:17.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 >> 172.21.15.1:0/3905467456 conn(0x7f8ba8078190 msgr2=0x7f8ba81062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:17.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 shutdown_connections 2024-04-26T22:48:17.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.148+0000 7f8bacee0700 1 -- 172.21.15.1:0/3905467456 wait complete. 2024-04-26T22:48:17.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:16 smithi186 ceph-mon[32774]: pgmap v308: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:17.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:16 smithi038 ceph-mon[32433]: pgmap v308: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/1706308797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 msgr2=0x7f9f481027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/1706308797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f481027a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c009990 tx=0x7f9f3c009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/1706308797 shutdown_connections 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/1706308797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f48102ce0 0x7f9f4810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/1706308797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f48108390 0x7f9f48108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/1706308797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f481027a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/1706308797 >> 172.21.15.1:0/1706308797 conn(0x7f9f480792d0 msgr2=0x7f9f480796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/1706308797 shutdown_connections 2024-04-26T22:48:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/1706308797 wait complete. 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 Processor -- start 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.425+0000 7f9f4e8a6700 1 -- start start 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4e8a6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f48104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4e8a6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f48102ce0 0x7f9f48104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4e8a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f48108390 0x7f9f48104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4e8a6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9f48112500 con 0x7f9f48102320 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4e8a6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f48112380 con 0x7f9f48108390 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4e8a6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f48112680 con 0x7f9f48102ce0 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f48104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f48104000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34824/0 (socket says 172.21.15.1:34824) 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 -- 172.21.15.1:0/3742913755 learned_addr learned my addr 172.21.15.1:0/3742913755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f4ce43700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f48108390 0x7f9f48104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f477fe700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f48102ce0 0x7f9f48104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 -- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f48102ce0 msgr2=0x7f9f48104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f48102ce0 0x7f9f48104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 -- 172.21.15.1:0/3742913755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f48108390 msgr2=0x7f9f48104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f48108390 0x7f9f48104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.426+0000 7f9f47fff700 1 -- 172.21.15.1:0/3742913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f481a8090 con 0x7f9f48102320 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.427+0000 7f9f47fff700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f48104000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800e9a0 tx=0x7f9f3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.427+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800ca10 con 0x7f9f48102320 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.427+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f481a8380 con 0x7f9f48102320 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.427+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f3800cbb0 con 0x7f9f48102320 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.427+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f38018680 con 0x7f9f48102320 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.427+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f481a8910 con 0x7f9f48102320 2024-04-26T22:48:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.428+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f48106410 con 0x7f9f48102320 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.429+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9f38010430 con 0x7f9f48102320 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f457fa700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9f300a3cd0 0x7f9f300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f477fe700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9f300a3cd0 0x7f9f300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9f380d1010 con 0x7f9f48102320 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f457fa700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] conn(0x7f9f300ad4b0 0x7f9f300af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 --> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f300aff80 con 0x7f9f300ad4b0 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f9f380d14f0 con 0x7f9f48102320 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f477fe700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9f300a3cd0 0x7f9f300a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9f480797d0 tx=0x7f9f3c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.430+0000 7f9f4ce43700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] conn(0x7f9f300ad4b0 0x7f9f300af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.431+0000 7f9f4ce43700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] conn(0x7f9f300ad4b0 0x7f9f300af8d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.432+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== osd.8 v2:172.21.15.186:6816/955115503 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f300aff80 con 0x7f9f300ad4b0 2024-04-26T22:48:17.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.448+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 --> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f48103a40 con 0x7f9f300ad4b0 2024-04-26T22:48:17.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f457fa700 1 -- 172.21.15.1:0/3742913755 <== osd.8 v2:172.21.15.186:6816/955115503 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9f48103a40 con 0x7f9f300ad4b0 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] conn(0x7f9f300ad4b0 msgr2=0x7f9f300af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] conn(0x7f9f300ad4b0 0x7f9f300af8d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9f300a3cd0 msgr2=0x7f9f300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9f300a3cd0 0x7f9f300a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9f480797d0 tx=0x7f9f3c00b260 comp rx=0 tx=0).stop 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 msgr2=0x7f9f48104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 --2- 172.21.15.1:0/3742913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f48102320 0x7f9f48104000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800e9a0 tx=0x7f9f3800ecb0 comp rx=0 tx=0).stop 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4ce43700 1 -- 172.21.15.1:0/3742913755 reap_dead start 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 shutdown_connections 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.449+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 >> 172.21.15.1:0/3742913755 conn(0x7f9f480792d0 msgr2=0x7f9f48101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.450+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 shutdown_connections 2024-04-26T22:48:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:17.450+0000 7f9f4e8a6700 1 -- 172.21.15.1:0/3742913755 wait complete. 2024-04-26T22:48:19.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:18 smithi001 ceph-mon[29435]: pgmap v309: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T22:48:19.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:18 smithi186 ceph-mon[32774]: pgmap v309: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T22:48:19.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:18 smithi038 ceph-mon[32433]: pgmap v309: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T22:48:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 -- 172.21.15.1:0/57130476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec102320 msgr2=0x7f84ec1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 --2- 172.21.15.1:0/57130476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec102320 0x7f84ec1027a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009990 tx=0x7f84e0009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 -- 172.21.15.1:0/57130476 shutdown_connections 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 --2- 172.21.15.1:0/57130476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84ec108390 0x7f84ec108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 --2- 172.21.15.1:0/57130476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec102ce0 0x7f84ec10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 --2- 172.21.15.1:0/57130476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec102320 0x7f84ec1027a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 -- 172.21.15.1:0/57130476 >> 172.21.15.1:0/57130476 conn(0x7f84ec0792d0 msgr2=0x7f84ec0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 -- 172.21.15.1:0/57130476 shutdown_connections 2024-04-26T22:48:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.973+0000 7f84f33f0700 1 -- 172.21.15.1:0/57130476 wait complete. 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 Processor -- start 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 -- start start 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84ec102320 0x7f84ec19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec102ce0 0x7f84ec19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec108390 0x7f84ec199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec112530 con 0x7f84ec108390 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1123b0 con 0x7f84ec102ce0 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f33f0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1126b0 con 0x7f84ec102320 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f198d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec108390 0x7f84ec199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f198d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec108390 0x7f84ec199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34832/0 (socket says 172.21.15.1:34832) 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f198d700 1 -- 172.21.15.1:0/1165847113 learned_addr learned my addr 172.21.15.1:0/1165847113 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f098b700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec102ce0 0x7f84ec19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f118c700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84ec102320 0x7f84ec19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.974+0000 7f84f198d700 1 -- 172.21.15.1:0/1165847113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84ec102320 msgr2=0x7f84ec19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f198d700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84ec102320 0x7f84ec19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f198d700 1 -- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec102ce0 msgr2=0x7f84ec19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f198d700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec102ce0 0x7f84ec19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f198d700 1 -- 172.21.15.1:0/1165847113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec19a0d0 con 0x7f84ec108390 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f198d700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec108390 0x7f84ec199970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84e400b970 tx=0x7f84e400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e400d860 con 0x7f84ec108390 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e400e430 con 0x7f84ec108390 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec19a3c0 con 0x7f84ec108390 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4017660 con 0x7f84ec108390 2024-04-26T22:48:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.975+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec1a6680 con 0x7f84ec108390 2024-04-26T22:48:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.976+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f84ec06b470 con 0x7f84ec108390 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.977+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f84e400da00 con 0x7f84ec108390 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84de7fc700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f84d80a3d50 0x7f84d80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84f118c700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f84d80a3d50 0x7f84d80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f84e40d0420 con 0x7f84ec108390 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84de7fc700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] conn(0x7f84d80ad530 0x7f84d80af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 --> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84e4011ec0 con 0x7f84d80ad530 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f84e40d07d0 con 0x7f84ec108390 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84f118c700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f84d80a3d50 0x7f84d80a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f84ec19aab0 tx=0x7f84e80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.978+0000 7f84f098b700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] conn(0x7f84d80ad530 0x7f84d80af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.979+0000 7f84f098b700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] conn(0x7f84d80ad530 0x7f84d80af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.979+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== osd.7 v2:172.21.15.38:6816/519750991 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84e4011ec0 con 0x7f84d80ad530 2024-04-26T22:48:19.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.996+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 --> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f84ec103a40 con 0x7f84d80ad530 2024-04-26T22:48:19.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84de7fc700 1 -- 172.21.15.1:0/1165847113 <== osd.7 v2:172.21.15.38:6816/519750991 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f84ec103a40 con 0x7f84d80ad530 2024-04-26T22:48:19.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] conn(0x7f84d80ad530 msgr2=0x7f84d80af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] conn(0x7f84d80ad530 0x7f84d80af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f84d80a3d50 msgr2=0x7f84d80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f84d80a3d50 0x7f84d80a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f84ec19aab0 tx=0x7f84e80073d0 comp rx=0 tx=0).stop 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec108390 msgr2=0x7f84ec199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 --2- 172.21.15.1:0/1165847113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec108390 0x7f84ec199970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84e400b970 tx=0x7f84e400bc80 comp rx=0 tx=0).stop 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f198d700 1 -- 172.21.15.1:0/1165847113 reap_dead start 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 shutdown_connections 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 >> 172.21.15.1:0/1165847113 conn(0x7f84ec0792d0 msgr2=0x7f84ec101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 shutdown_connections 2024-04-26T22:48:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:19.997+0000 7f84f33f0700 1 -- 172.21.15.1:0/1165847113 wait complete. 2024-04-26T22:48:20.749 INFO:teuthology.orchestra.run.smithi001.stdout:253403070501 2024-04-26T22:48:20.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 -- 172.21.15.1:0/3036783771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801094e0 msgr2=0x7fd8801098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 --2- 172.21.15.1:0/3036783771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801094e0 0x7fd8801098c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009990 tx=0x7fd87c009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 -- 172.21.15.1:0/3036783771 shutdown_connections 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 --2- 172.21.15.1:0/3036783771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880103e60 0x7fd8801102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 --2- 172.21.15.1:0/3036783771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801094e0 0x7fd8801098c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 --2- 172.21.15.1:0/3036783771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8801034a0 0x7fd880103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 -- 172.21.15.1:0/3036783771 >> 172.21.15.1:0/3036783771 conn(0x7fd88007a590 msgr2=0x7fd88007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 -- 172.21.15.1:0/3036783771 shutdown_connections 2024-04-26T22:48:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 -- 172.21.15.1:0/3036783771 wait complete. 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 Processor -- start 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.196+0000 7fd887305700 1 -- start start 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd887305700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801034a0 0x7fd880075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd887305700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880103e60 0x7fd880075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd887305700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8801094e0 0x7fd8800761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd887305700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801135e0 con 0x7fd8801034a0 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd887305700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd880113460 con 0x7fd8801094e0 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd887305700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd880113760 con 0x7fd880103e60 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801034a0 0x7fd880075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801034a0 0x7fd880075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34850/0 (socket says 172.21.15.1:34850) 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 -- 172.21.15.1:0/1107208525 learned_addr learned my addr 172.21.15.1:0/1107208525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8848a0700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880103e60 0x7fd880075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8858a2700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8801094e0 0x7fd8800761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 -- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880103e60 msgr2=0x7fd880075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880103e60 0x7fd880075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 -- 172.21.15.1:0/1107208525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8801094e0 msgr2=0x7fd8800761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8801094e0 0x7fd8800761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 -- 172.21.15.1:0/1107208525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8801a7f90 con 0x7fd8801034a0 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8850a1700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801034a0 0x7fd880075720 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd87c000c00 tx=0x7fd87c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c027070 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.197+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c003fe0 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.198+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8801a8220 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.198+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c022e30 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.198+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8801a8750 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.198+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd880063b30 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd87c02a070 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8767fc700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd86c0a3d50 0x7fd86c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8848a0700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd86c0a3d50 0x7fd86c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd87c0d1560 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8767fc700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] conn(0x7fd86c0ad530 0x7fd86c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 --> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd86c0b0000 con 0x7fd86c0ad530 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.200+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fd87c0d1a40 con 0x7fd8801034a0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.201+0000 7fd8848a0700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd86c0a3d50 0x7fd86c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd88007ac00 tx=0x7fd87000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.201+0000 7fd8858a2700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] conn(0x7fd86c0ad530 0x7fd86c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.201+0000 7fd8858a2700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] conn(0x7fd86c0ad530 0x7fd86c0af950 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.202+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== osd.5 v2:172.21.15.186:6808/1470594715 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd86c0b0000 con 0x7fd86c0ad530 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.218+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 --> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd880063f30 con 0x7fd86c0ad530 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd8767fc700 1 -- 172.21.15.1:0/1107208525 <== osd.5 v2:172.21.15.186:6808/1470594715 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd880063f30 con 0x7fd86c0ad530 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] conn(0x7fd86c0ad530 msgr2=0x7fd86c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] conn(0x7fd86c0ad530 0x7fd86c0af950 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd86c0a3d50 msgr2=0x7fd86c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd86c0a3d50 0x7fd86c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd88007ac00 tx=0x7fd87000a380 comp rx=0 tx=0).stop 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801034a0 msgr2=0x7fd880075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 --2- 172.21.15.1:0/1107208525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8801034a0 0x7fd880075720 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd87c000c00 tx=0x7fd87c0047b0 comp rx=0 tx=0).stop 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd8858a2700 1 -- 172.21.15.1:0/1107208525 reap_dead start 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 shutdown_connections 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 >> 172.21.15.1:0/1107208525 conn(0x7fd88007a590 msgr2=0x7fd880102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.219+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 shutdown_connections 2024-04-26T22:48:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:20.220+0000 7fd887305700 1 -- 172.21.15.1:0/1107208525 wait complete. 2024-04-26T22:48:21.038 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:20 smithi001 ceph-mon[29435]: pgmap v310: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:21.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:20 smithi186 ceph-mon[32774]: pgmap v310: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:21.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:20 smithi038 ceph-mon[32433]: pgmap v310: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:21.388 INFO:teuthology.orchestra.run.smithi001.stdout:309237645338 2024-04-26T22:48:21.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T22:48:21.989 INFO:teuthology.orchestra.run.smithi001.stdout:163208757301 2024-04-26T22:48:21.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T22:48:22.635 INFO:teuthology.orchestra.run.smithi001.stdout:141733920825 2024-04-26T22:48:22.635 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T22:48:23.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:22 smithi186 ceph-mon[32774]: pgmap v311: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:48:23.275 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:22 smithi001 ceph-mon[29435]: pgmap v311: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:48:23.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:22 smithi038 ceph-mon[32433]: pgmap v311: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:48:23.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3198197830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d4103e60 msgr2=0x7f35d41102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:23.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3198197830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d4103e60 0x7f35d41102b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00b0a0 tx=0x7f35cc00b3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3198197830 shutdown_connections 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3198197830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35d41094e0 0x7f35d41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3198197830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d4103e60 0x7f35d41102b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3198197830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d41034a0 0x7f35d4103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3198197830 >> 172.21.15.1:0/3198197830 conn(0x7f35d407a590 msgr2=0x7f35d407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3198197830 shutdown_connections 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.635+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3198197830 wait complete. 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 Processor -- start 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 -- start start 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35d41034a0 0x7f35d4075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d4103e60 0x7f35d4075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d41094e0 0x7f35d40761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f35d4113660 con 0x7f35d41094e0 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35d41134e0 con 0x7f35d4103e60 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d9a3f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f35d41137e0 con 0x7f35d41034a0 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d41094e0 0x7f35d40761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d41094e0 0x7f35d40761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35914/0 (socket says 172.21.15.1:35914) 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 -- 172.21.15.1:0/3679374440 learned_addr learned my addr 172.21.15.1:0/3679374440 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d27fc700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d4103e60 0x7f35d4075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d2ffd700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35d41034a0 0x7f35d4075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 -- 172.21.15.1:0/3679374440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35d41034a0 msgr2=0x7f35d4075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35d41034a0 0x7f35d4075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 -- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d4103e60 msgr2=0x7f35d4075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d4103e60 0x7f35d4075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d37fe700 1 -- 172.21.15.1:0/3679374440 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d41a7fa0 con 0x7f35d41094e0 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.636+0000 7f35d27fc700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d4103e60 0x7f35d4075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.637+0000 7f35d37fe700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d41094e0 0x7f35d40761a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0062a0 tx=0x7f35cc004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.637+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc021070 con 0x7f35d41094e0 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.637+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d41a8230 con 0x7f35d41094e0 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.637+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35cc007d00 con 0x7f35d41094e0 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.637+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc01d450 con 0x7f35d41094e0 2024-04-26T22:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.637+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d41a8770 con 0x7f35d41094e0 2024-04-26T22:48:23.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.638+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35d4063b30 con 0x7f35d41094e0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.639+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f35cc004000 con 0x7f35d41094e0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d8a3d700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f35bc0a3cd0 0x7f35bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d2ffd700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f35bc0a3cd0 0x7f35bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f35cc00f020 con 0x7f35d41094e0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d8a3d700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] conn(0x7f35bc0ad4b0 0x7f35bc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 --> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35cc0020b0 con 0x7f35bc0ad4b0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f35cc0d2200 con 0x7f35d41094e0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d2ffd700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f35bc0a3cd0 0x7f35bc0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f35c4005ea0 tx=0x7f35c4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.640+0000 7f35d27fc700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] conn(0x7f35bc0ad4b0 0x7f35bc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.641+0000 7f35d27fc700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] conn(0x7f35bc0ad4b0 0x7f35bc0af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.641+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== osd.1 v2:172.21.15.38:6800/156758275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35cc0020b0 con 0x7f35bc0ad4b0 2024-04-26T22:48:23.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.658+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 --> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f35d4063fb0 con 0x7f35bc0ad4b0 2024-04-26T22:48:23.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.658+0000 7f35d8a3d700 1 -- 172.21.15.1:0/3679374440 <== osd.1 v2:172.21.15.38:6800/156758275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f35d4063fb0 con 0x7f35bc0ad4b0 2024-04-26T22:48:23.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] conn(0x7f35bc0ad4b0 msgr2=0x7f35bc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:23.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] conn(0x7f35bc0ad4b0 0x7f35bc0af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f35bc0a3cd0 msgr2=0x7f35bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f35bc0a3cd0 0x7f35bc0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f35c4005ea0 tx=0x7f35c4005e30 comp rx=0 tx=0).stop 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d41094e0 msgr2=0x7f35d40761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 --2- 172.21.15.1:0/3679374440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35d41094e0 0x7f35d40761a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0062a0 tx=0x7f35cc004ed0 comp rx=0 tx=0).stop 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d37fe700 1 -- 172.21.15.1:0/3679374440 reap_dead start 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 shutdown_connections 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 >> 172.21.15.1:0/3679374440 conn(0x7f35d407a590 msgr2=0x7f35d4102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 shutdown_connections 2024-04-26T22:48:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:23.659+0000 7f35d9a3f700 1 -- 172.21.15.1:0/3679374440 wait complete. 2024-04-26T22:48:24.101 INFO:teuthology.orchestra.run.smithi001.stdout:219043332139 2024-04-26T22:48:24.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T22:48:24.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 -- 172.21.15.1:0/666087812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 msgr2=0x7f34b0103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 --2- 172.21.15.1:0/666087812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b0103920 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009990 tx=0x7f34a0009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 -- 172.21.15.1:0/666087812 shutdown_connections 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 --2- 172.21.15.1:0/666087812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34b01094e0 0x7f34b01098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 --2- 172.21.15.1:0/666087812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0103e60 0x7f34b01102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 --2- 172.21.15.1:0/666087812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b0103920 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 -- 172.21.15.1:0/666087812 >> 172.21.15.1:0/666087812 conn(0x7f34b007a590 msgr2=0x7f34b007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 -- 172.21.15.1:0/666087812 shutdown_connections 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 -- 172.21.15.1:0/666087812 wait complete. 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.927+0000 7f34b8482700 1 Processor -- start 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 -- start start 2024-04-26T22:48:24.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0103e60 0x7f34b019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34b01094e0 0x7f34b01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f34b0113660 con 0x7f34b01034a0 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01134e0 con 0x7f34b0103e60 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b8482700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01137e0 con 0x7f34b01094e0 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35920/0 (socket says 172.21.15.1:35920) 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 -- 172.21.15.1:0/468087469 learned_addr learned my addr 172.21.15.1:0/468087469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b6a1f700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34b01094e0 0x7f34b01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b5a1d700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0103e60 0x7f34b019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 -- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34b01094e0 msgr2=0x7f34b01999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34b01094e0 0x7f34b01999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 -- 172.21.15.1:0/468087469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0103e60 msgr2=0x7f34b019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0103e60 0x7f34b019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 -- 172.21.15.1:0/468087469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b019a120 con 0x7f34b01034a0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.928+0000 7f34b621e700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b019f800 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b530 tx=0x7f34ac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.929+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac015020 con 0x7f34b01034a0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.929+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34ac011ce0 con 0x7f34b01034a0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.929+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b019a410 con 0x7f34b01034a0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.929+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac010dd0 con 0x7f34b01034a0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.929+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b01a66d0 con 0x7f34b01034a0 2024-04-26T22:48:24.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.929+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34b0063b30 con 0x7f34b01034a0 2024-04-26T22:48:24.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.931+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f34ac018020 con 0x7f34b01034a0 2024-04-26T22:48:24.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.931+0000 7f34a77fe700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f349c0a3d50 0x7f349c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34b5a1d700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f349c0a3d50 0x7f349c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f34ac0d0100 con 0x7f34b01034a0 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34a77fe700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] conn(0x7f349c0ad530 0x7f349c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 --> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f349c0b0000 con 0x7f349c0ad530 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f34ac0d04b0 con 0x7f34b01034a0 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34b5a1d700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f349c0a3d50 0x7f349c0a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34a000b5c0 tx=0x7f34a0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34b6a1f700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] conn(0x7f349c0ad530 0x7f349c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.932+0000 7f34b6a1f700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] conn(0x7f349c0ad530 0x7f349c0af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.933+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== osd.11 v2:172.21.15.186:6824/63751610 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f349c0b0000 con 0x7f349c0ad530 2024-04-26T22:48:24.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.950+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 --> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34b0063fb0 con 0x7f349c0ad530 2024-04-26T22:48:24.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.950+0000 7f34a77fe700 1 -- 172.21.15.1:0/468087469 <== osd.11 v2:172.21.15.186:6824/63751610 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f34b0063fb0 con 0x7f349c0ad530 2024-04-26T22:48:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] conn(0x7f349c0ad530 msgr2=0x7f349c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] conn(0x7f349c0ad530 0x7f349c0af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f349c0a3d50 msgr2=0x7f349c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f349c0a3d50 0x7f349c0a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34a000b5c0 tx=0x7f34a0005960 comp rx=0 tx=0).stop 2024-04-26T22:48:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 msgr2=0x7f34b019f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 --2- 172.21.15.1:0/468087469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34b01034a0 0x7f34b019f800 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b530 tx=0x7f34ac00b840 comp rx=0 tx=0).stop 2024-04-26T22:48:24.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b6a1f700 1 -- 172.21.15.1:0/468087469 reap_dead start 2024-04-26T22:48:24.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 shutdown_connections 2024-04-26T22:48:24.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 >> 172.21.15.1:0/468087469 conn(0x7f34b007a590 msgr2=0x7f34b0102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:24.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 shutdown_connections 2024-04-26T22:48:24.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:24.951+0000 7f34b8482700 1 -- 172.21.15.1:0/468087469 wait complete. 2024-04-26T22:48:25.054 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:24 smithi001 ceph-mon[29435]: pgmap v312: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:25.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:24 smithi186 ceph-mon[32774]: pgmap v312: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:25.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 -- 172.21.15.1:0/1389286217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c103e30 msgr2=0x7f190c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:25.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 --2- 172.21.15.1:0/1389286217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c103e30 0x7f190c110290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1904009990 tx=0x7f1904009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 -- 172.21.15.1:0/1389286217 shutdown_connections 2024-04-26T22:48:25.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 --2- 172.21.15.1:0/1389286217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f190c1094c0 0x7f190c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 --2- 172.21.15.1:0/1389286217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c103e30 0x7f190c110290 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 --2- 172.21.15.1:0/1389286217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c103470 0x7f190c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 -- 172.21.15.1:0/1389286217 >> 172.21.15.1:0/1389286217 conn(0x7f190c07a580 msgr2=0x7f190c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 -- 172.21.15.1:0/1389286217 shutdown_connections 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.268+0000 7f1910dfd700 1 -- 172.21.15.1:0/1389286217 wait complete. 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.269+0000 7f1910dfd700 1 Processor -- start 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.269+0000 7f1910dfd700 1 -- start start 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f190c103470 0x7f190c19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c103e30 0x7f190c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c1094c0 0x7f190c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f190c1135c0 con 0x7f190c1094c0 2024-04-26T22:48:25.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f190c113440 con 0x7f190c103e30 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f190c113740 con 0x7f190c103470 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c1094c0 0x7f190c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c1094c0 0x7f190c199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35932/0 (socket says 172.21.15.1:35932) 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 -- 172.21.15.1:0/2327382467 learned_addr learned my addr 172.21.15.1:0/2327382467 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1909d9b700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c103e30 0x7f190c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190a59c700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f190c103470 0x7f190c19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 -- 172.21.15.1:0/2327382467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f190c103470 msgr2=0x7f190c19f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f190c103470 0x7f190c19f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 -- 172.21.15.1:0/2327382467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c103e30 msgr2=0x7f190c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c103e30 0x7f190c19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 -- 172.21.15.1:0/2327382467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c199fe0 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f190ad9d700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c1094c0 0x7f190c199880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f19040057c0 tx=0x7f19040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904027070 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190c19a270 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1904003f20 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904022ce0 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.270+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f190c1a6590 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.271+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f190c063b30 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.273+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f190402a070 con 0x7f190c1094c0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.273+0000 7f19037fe700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f18f80a3cd0 0x7f18f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.273+0000 7f190a59c700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f18f80a3cd0 0x7f18f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.273+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f19040d1dd0 con 0x7f190c1094c0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.274+0000 7f19037fe700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] conn(0x7f18f80ad4b0 0x7f18f80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.274+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 --> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18f80aff80 con 0x7f18f80ad4b0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.274+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f19040d22b0 con 0x7f190c1094c0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.274+0000 7f1909d9b700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] conn(0x7f18f80ad4b0 0x7f18f80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.274+0000 7f190a59c700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f18f80a3cd0 0x7f18f80a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f190c19a9c0 tx=0x7f18fc009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.274+0000 7f1909d9b700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] conn(0x7f18f80ad4b0 0x7f18f80af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.275+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== osd.6 v2:172.21.15.1:6818/244390713 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f18f80aff80 con 0x7f18f80ad4b0 2024-04-26T22:48:25.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.291+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 --> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f190c063f30 con 0x7f18f80ad4b0 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.291+0000 7f19037fe700 1 -- 172.21.15.1:0/2327382467 <== osd.6 v2:172.21.15.1:6818/244390713 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f190c063f30 con 0x7f18f80ad4b0 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] conn(0x7f18f80ad4b0 msgr2=0x7f18f80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] conn(0x7f18f80ad4b0 0x7f18f80af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f18f80a3cd0 msgr2=0x7f18f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f18f80a3cd0 0x7f18f80a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f190c19a9c0 tx=0x7f18fc009040 comp rx=0 tx=0).stop 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c1094c0 msgr2=0x7f190c199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 --2- 172.21.15.1:0/2327382467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190c1094c0 0x7f190c199880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f19040057c0 tx=0x7f19040057f0 comp rx=0 tx=0).stop 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f190ad9d700 1 -- 172.21.15.1:0/2327382467 reap_dead start 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 shutdown_connections 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 >> 172.21.15.1:0/2327382467 conn(0x7f190c07a580 msgr2=0x7f190c1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 shutdown_connections 2024-04-26T22:48:25.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:25.292+0000 7f1910dfd700 1 -- 172.21.15.1:0/2327382467 wait complete. 2024-04-26T22:48:25.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:24 smithi038 ceph-mon[32433]: pgmap v312: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:25.957 INFO:teuthology.orchestra.run.smithi001.stdout:197568495665 2024-04-26T22:48:25.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T22:48:26.854 INFO:teuthology.orchestra.run.smithi001.stdout:55834574922 2024-04-26T22:48:26.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T22:48:27.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:26 smithi001 ceph-mon[29435]: pgmap v313: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-26T22:48:27.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:26 smithi038 ceph-mon[32433]: pgmap v313: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-26T22:48:27.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:26 smithi186 ceph-mon[32774]: pgmap v313: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-26T22:48:28.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 -- 172.21.15.1:0/2707176151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 msgr2=0x7f88f0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:28.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 --2- 172.21.15.1:0/2707176151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 0x7f88f0110290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f88e800b0a0 tx=0x7f88e800b3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 -- 172.21.15.1:0/2707176151 shutdown_connections 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 --2- 172.21.15.1:0/2707176151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88f01094c0 0x7f88f01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 --2- 172.21.15.1:0/2707176151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 0x7f88f0110290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 --2- 172.21.15.1:0/2707176151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 0x7f88f01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 -- 172.21.15.1:0/2707176151 >> 172.21.15.1:0/2707176151 conn(0x7f88f007a580 msgr2=0x7f88f007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 -- 172.21.15.1:0/2707176151 shutdown_connections 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.136+0000 7f88f524e700 1 -- 172.21.15.1:0/2707176151 wait complete. 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 Processor -- start 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 -- start start 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 0x7f88f019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 0x7f88f019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88f01094c0 0x7f88f0199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f88f0113640 con 0x7f88f0103470 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01134c0 con 0x7f88f0103e30 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88f524e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01137c0 con 0x7f88f01094c0 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.137+0000 7f88eed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 0x7f88f019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 0x7f88f019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35960/0 (socket says 172.21.15.1:35960) 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 -- 172.21.15.1:0/1702189253 learned_addr learned my addr 172.21.15.1:0/1702189253 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88ee59c700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 0x7f88f019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88ef59e700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88f01094c0 0x7f88f0199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 -- 172.21.15.1:0/1702189253 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88f01094c0 msgr2=0x7f88f0199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88f01094c0 0x7f88f0199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 -- 172.21.15.1:0/1702189253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 msgr2=0x7f88f019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 0x7f88f019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 -- 172.21.15.1:0/1702189253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f019a0c0 con 0x7f88f0103470 2024-04-26T22:48:28.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88ee59c700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88f0103e30 0x7f88f019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88eed9d700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 0x7f88f019f750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b530 tx=0x7f88e000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0015020 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f019a3b0 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88e0011ce0 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0010dd0 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.138+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f01a6620 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.139+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88f0063b70 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.140+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f88e0018020 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88d7fff700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f88d80a3cd0 0x7f88d80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88ee59c700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f88d80a3cd0 0x7f88d80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f88e00d0230 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88d7fff700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] conn(0x7f88d80ad4b0 0x7f88d80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 --> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88e0018ec0 con 0x7f88d80ad4b0 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f88e00d05e0 con 0x7f88f0103470 2024-04-26T22:48:28.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88ef59e700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] conn(0x7f88d80ad4b0 0x7f88d80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:28.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.141+0000 7f88ee59c700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f88d80a3cd0 0x7f88d80a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88f007ab80 tx=0x7f88e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:28.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.142+0000 7f88ef59e700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] conn(0x7f88d80ad4b0 0x7f88d80af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:28.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.142+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== osd.0 v2:172.21.15.1:6802/1864154398 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88e0018ec0 con 0x7f88d80ad4b0 2024-04-26T22:48:28.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.160+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 --> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88f0063ff0 con 0x7f88d80ad4b0 2024-04-26T22:48:28.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88d7fff700 1 -- 172.21.15.1:0/1702189253 <== osd.0 v2:172.21.15.1:6802/1864154398 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f88f0063ff0 con 0x7f88d80ad4b0 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] conn(0x7f88d80ad4b0 msgr2=0x7f88d80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88f524e700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] conn(0x7f88d80ad4b0 0x7f88d80af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f88d80a3cd0 msgr2=0x7f88d80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88f524e700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f88d80a3cd0 0x7f88d80a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88f007ab80 tx=0x7f88e4009260 comp rx=0 tx=0).stop 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 msgr2=0x7f88f019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88f524e700 1 --2- 172.21.15.1:0/1702189253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88f0103470 0x7f88f019f750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b530 tx=0x7f88e000b840 comp rx=0 tx=0).stop 2024-04-26T22:48:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.161+0000 7f88ef59e700 1 -- 172.21.15.1:0/1702189253 reap_dead start 2024-04-26T22:48:28.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.162+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 shutdown_connections 2024-04-26T22:48:28.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.162+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 >> 172.21.15.1:0/1702189253 conn(0x7f88f007a580 msgr2=0x7f88f01021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:28.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.162+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 shutdown_connections 2024-04-26T22:48:28.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.162+0000 7f88f524e700 1 -- 172.21.15.1:0/1702189253 wait complete. 2024-04-26T22:48:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 -- 172.21.15.1:0/332208937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 msgr2=0x7fcd7c1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/332208937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c1027a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcd64009990 tx=0x7fcd64009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 -- 172.21.15.1:0/332208937 shutdown_connections 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/332208937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd7c108390 0x7fcd7c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/332208937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd7c102ce0 0x7fcd7c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/332208937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c1027a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 -- 172.21.15.1:0/332208937 >> 172.21.15.1:0/332208937 conn(0x7fcd7c0792d0 msgr2=0x7fcd7c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 -- 172.21.15.1:0/332208937 shutdown_connections 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.041+0000 7fcd81ab7700 1 -- 172.21.15.1:0/332208937 wait complete. 2024-04-26T22:48:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 Processor -- start 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 -- start start 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd7c102ce0 0x7fcd7c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd7c108390 0x7fcd7c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c1124d0 con 0x7fcd7c102320 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c112350 con 0x7fcd7c102ce0 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.042+0000 7fcd81ab7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c112650 con 0x7fcd7c108390 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35946/0 (socket says 172.21.15.1:35946) 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 -- 172.21.15.1:0/262742850 learned_addr learned my addr 172.21.15.1:0/262742850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd72ffd700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd7c102ce0 0x7fcd7c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7bfff700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd7c108390 0x7fcd7c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 -- 172.21.15.1:0/262742850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd7c108390 msgr2=0x7fcd7c1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd7c108390 0x7fcd7c1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 -- 172.21.15.1:0/262742850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd7c102ce0 msgr2=0x7fcd7c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd7c102ce0 0x7fcd7c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 -- 172.21.15.1:0/262742850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c19a040 con 0x7fcd7c102320 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7bfff700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd7c108390 0x7fcd7c1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd72ffd700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd7c102ce0 0x7fcd7c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd7b7fe700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c19f720 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00c970 tx=0x7fcd6c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c007aa0 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c19a330 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd6c004ce0 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c005510 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.043+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c1a65f0 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.044+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcd7c106490 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.046+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fcd6c007c40 con 0x7fcd7c102320 2024-04-26T22:48:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.046+0000 7fcd797fa700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcd680a3cd0 0x7fcd680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.046+0000 7fcd72ffd700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcd680a3cd0 0x7fcd680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.046+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcd6c0d0560 con 0x7fcd7c102320 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.046+0000 7fcd797fa700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] conn(0x7fcd680ad4b0 0x7fcd680af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.046+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 --> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd6c00eec0 con 0x7fcd680ad4b0 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.047+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fcd6c0d0a40 con 0x7fcd7c102320 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.047+0000 7fcd7bfff700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] conn(0x7fcd680ad4b0 0x7fcd680af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.047+0000 7fcd72ffd700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcd680a3cd0 0x7fcd680a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcd6400b2e0 tx=0x7fcd6401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:29.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.047+0000 7fcd7bfff700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] conn(0x7fcd680ad4b0 0x7fcd680af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.048+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== osd.3 v2:172.21.15.1:6810/10840892 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcd6c00eec0 con 0x7fcd680ad4b0 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.066+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 --> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcd7c103a40 con 0x7fcd680ad4b0 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd797fa700 1 -- 172.21.15.1:0/262742850 <== osd.3 v2:172.21.15.1:6810/10840892 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcd7c103a40 con 0x7fcd680ad4b0 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] conn(0x7fcd680ad4b0 msgr2=0x7fcd680af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] conn(0x7fcd680ad4b0 0x7fcd680af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcd680a3cd0 msgr2=0x7fcd680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcd680a3cd0 0x7fcd680a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcd6400b2e0 tx=0x7fcd6401a040 comp rx=0 tx=0).stop 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 msgr2=0x7fcd7c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 --2- 172.21.15.1:0/262742850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd7c102320 0x7fcd7c19f720 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00c970 tx=0x7fcd6c00cc80 comp rx=0 tx=0).stop 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd7bfff700 1 -- 172.21.15.1:0/262742850 reap_dead start 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 shutdown_connections 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.067+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 >> 172.21.15.1:0/262742850 conn(0x7fcd7c0792d0 msgr2=0x7fcd7c101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:29.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.068+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 shutdown_connections 2024-04-26T22:48:29.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.068+0000 7fcd81ab7700 1 -- 172.21.15.1:0/262742850 wait complete. 2024-04-26T22:48:29.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:28 smithi186 ceph-mon[32774]: pgmap v314: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:29.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:28 smithi001 ceph-mon[29435]: pgmap v314: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:29.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:28 smithi038 ceph-mon[32433]: pgmap v314: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:29.711 INFO:teuthology.orchestra.run.smithi001.stdout:107374182465 2024-04-26T22:48:29.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T22:48:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-26T22:48:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T22:48:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-26T22:48:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-26T22:48:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-26T22:48:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-26T22:48:30.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:29 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T22:48:30.171 INFO:teuthology.orchestra.run.smithi001.stdout:343597383703 2024-04-26T22:48:30.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-26T22:48:30.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:29 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T22:48:30.306 INFO:teuthology.orchestra.run.smithi001.stdout:34359738446 2024-04-26T22:48:30.306 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-26T22:48:30.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:29 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T22:48:30.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 -- 172.21.15.1:0/4133100948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 msgr2=0x7ff7f81102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:30.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 --2- 172.21.15.1:0/4133100948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f81102e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000b0a0 tx=0x7ff7f000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 -- 172.21.15.1:0/4133100948 shutdown_connections 2024-04-26T22:48:30.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 --2- 172.21.15.1:0/4133100948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f81094a0 0x7ff7f8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 --2- 172.21.15.1:0/4133100948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f81102e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 --2- 172.21.15.1:0/4133100948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7f806b130 0x7ff7f806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.747+0000 7ff7fd914700 1 -- 172.21.15.1:0/4133100948 >> 172.21.15.1:0/4133100948 conn(0x7ff7f80792d0 msgr2=0x7ff7f80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.748+0000 7ff7fd914700 1 -- 172.21.15.1:0/4133100948 shutdown_connections 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.748+0000 7ff7fd914700 1 -- 172.21.15.1:0/4133100948 wait complete. 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.748+0000 7ff7fd914700 1 Processor -- start 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 -- start start 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f806b130 0x7ff7f8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7f81094a0 0x7ff7f819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81135c0 con 0x7ff7f806bad0 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8113440 con 0x7ff7f81094a0 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7fd914700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8113740 con 0x7ff7f806b130 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35982/0 (socket says 172.21.15.1:35982) 2024-04-26T22:48:30.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 -- 172.21.15.1:0/3077976920 learned_addr learned my addr 172.21.15.1:0/3077976920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f77fe700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7f81094a0 0x7ff7f819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f6ffd700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f806b130 0x7ff7f8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 -- 172.21.15.1:0/3077976920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f806b130 msgr2=0x7ff7f8199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7f806b130 0x7ff7f8199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 -- 172.21.15.1:0/3077976920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7f81094a0 msgr2=0x7ff7f819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7f81094a0 0x7ff7f819e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 -- 172.21.15.1:0/3077976920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f819eb20 con 0x7ff7f806bad0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.749+0000 7ff7f67fc700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f819a0b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00d950 tx=0x7ff7ec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.750+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec016070 con 0x7ff7f806bad0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.750+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f819ee10 con 0x7ff7f806bad0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.750+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7ec009df0 con 0x7ff7f806bad0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.750+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00f3c0 con 0x7ff7f806bad0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.750+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81a66d0 con 0x7ff7f806bad0 2024-04-26T22:48:30.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.750+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7f8063af0 con 0x7ff7f806bad0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.752+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff7ec019070 con 0x7ff7f806bad0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.752+0000 7ff7fc912700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff7e00a3cd0 0x7ff7e00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.752+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff7ec0d0490 con 0x7ff7f806bad0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.752+0000 7ff7f6ffd700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff7e00a3cd0 0x7ff7e00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.753+0000 7ff7fc912700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] conn(0x7ff7e00ad4b0 0x7ff7e00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.753+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 --> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7e00aff80 con 0x7ff7e00ad4b0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.753+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7ff7ec0d0970 con 0x7ff7f806bad0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.753+0000 7ff7f77fe700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] conn(0x7ff7e00ad4b0 0x7ff7e00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.753+0000 7ff7f6ffd700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff7e00a3cd0 0x7ff7e00a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800bbb0 tx=0x7ff7e800b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.753+0000 7ff7f77fe700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] conn(0x7ff7e00ad4b0 0x7ff7e00af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.754+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== osd.9 v2:172.21.15.1:6826/2190648434 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7e00aff80 con 0x7ff7e00ad4b0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 --> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7f8063ef0 con 0x7ff7e00ad4b0 2024-04-26T22:48:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fc912700 1 -- 172.21.15.1:0/3077976920 <== osd.9 v2:172.21.15.1:6826/2190648434 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff7f8063ef0 con 0x7ff7e00ad4b0 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] conn(0x7ff7e00ad4b0 msgr2=0x7ff7e00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] conn(0x7ff7e00ad4b0 0x7ff7e00af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff7e00a3cd0 msgr2=0x7ff7e00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7ff7e00a3cd0 0x7ff7e00a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800bbb0 tx=0x7ff7e800b590 comp rx=0 tx=0).stop 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 msgr2=0x7ff7f819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.771+0000 7ff7fd914700 1 --2- 172.21.15.1:0/3077976920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7f806bad0 0x7ff7f819a0b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00d950 tx=0x7ff7ec00dc60 comp rx=0 tx=0).stop 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.772+0000 7ff7f77fe700 1 -- 172.21.15.1:0/3077976920 reap_dead start 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.772+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 shutdown_connections 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.772+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 >> 172.21.15.1:0/3077976920 conn(0x7ff7f80792d0 msgr2=0x7ff7f8100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.772+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 shutdown_connections 2024-04-26T22:48:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:28.772+0000 7ff7fd914700 1 -- 172.21.15.1:0/3077976920 wait complete. 2024-04-26T22:48:30.683 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:30.683 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:30.684 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:30.684 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:30.687 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: pgmap v315: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T22:48:31.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:30 smithi001 ceph-mon[29435]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T22:48:31.173 INFO:teuthology.orchestra.run.smithi001.stdout:287762808866 2024-04-26T22:48:31.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: pgmap v315: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-26T22:48:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-26T22:48:31.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T22:48:31.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:30 smithi186 ceph-mon[32774]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T22:48:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: pgmap v315: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-26T22:48:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-26T22:48:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T22:48:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-26T22:48:31.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-26T22:48:31.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-26T22:48:31.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-26T22:48:31.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T22:48:31.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:30 smithi038 ceph-mon[32433]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T22:48:32.094 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:31 smithi001 ceph-mon[29435]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T22:48:32.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:31 smithi186 ceph-mon[32774]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T22:48:32.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:31 smithi038 ceph-mon[32433]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T22:48:33.025 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:32 smithi001 ceph-mon[29435]: pgmap v318: 97 pgs: 4 peering, 93 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T22:48:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:32 smithi186 ceph-mon[32774]: pgmap v318: 97 pgs: 4 peering, 93 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T22:48:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:32 smithi038 ceph-mon[32433]: pgmap v318: 97 pgs: 4 peering, 93 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T22:48:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:33.620 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:33.621 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:35.219 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:34 smithi001 ceph-mon[29435]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:35.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:34 smithi186 ceph-mon[32774]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:34 smithi038 ceph-mon[32433]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:37.087 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:36 smithi001 ceph-mon[29435]: pgmap v320: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:37.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:36 smithi186 ceph-mon[32774]: pgmap v320: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:36 smithi038 ceph-mon[32433]: pgmap v320: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:39.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:38 smithi001 ceph-mon[29435]: pgmap v321: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:39.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:38 smithi186 ceph-mon[32774]: pgmap v321: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:38 smithi038 ceph-mon[32433]: pgmap v321: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:39.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/2735381644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a01034a0 msgr2=0x7fd2a0103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:39.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/2735381644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a01034a0 0x7fd2a0103920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009990 tx=0x7fd28c009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/2735381644 shutdown_connections 2024-04-26T22:48:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/2735381644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a01094e0 0x7fd2a01098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/2735381644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2a0103e60 0x7fd2a01102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/2735381644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a01034a0 0x7fd2a0103920 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/2735381644 >> 172.21.15.1:0/2735381644 conn(0x7fd2a007a590 msgr2=0x7fd2a007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/2735381644 shutdown_connections 2024-04-26T22:48:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.389+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/2735381644 wait complete. 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 Processor -- start 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 -- start start 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2a01034a0 0x7fd2a019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 0x7fd2a019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a01094e0 0x7fd2a01a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0113600 con 0x7fd2a0103e60 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0113480 con 0x7fd2a01034a0 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a6f4e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0113780 con 0x7fd2a01094e0 2024-04-26T22:48:39.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 0x7fd2a019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 0x7fd2a019c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36504/0 (socket says 172.21.15.1:36504) 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 -- 172.21.15.1:0/374056674 learned_addr learned my addr 172.21.15.1:0/374056674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a4cea700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2a01034a0 0x7fd2a019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd2a54eb700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a01094e0 0x7fd2a01a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 -- 172.21.15.1:0/374056674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a01094e0 msgr2=0x7fd2a01a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a01094e0 0x7fd2a01a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 -- 172.21.15.1:0/374056674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2a01034a0 msgr2=0x7fd2a019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2a01034a0 0x7fd2a019bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.390+0000 7fd29ffff700 1 -- 172.21.15.1:0/374056674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a01a0d40 con 0x7fd2a0103e60 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.391+0000 7fd29ffff700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 0x7fd2a019c270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd28c000c00 tx=0x7fd28c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.391+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c027070 con 0x7fd2a0103e60 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.391+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd28c003f20 con 0x7fd2a0103e60 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.391+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a01a0fd0 con 0x7fd2a0103e60 2024-04-26T22:48:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.391+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c022cf0 con 0x7fd2a0103e60 2024-04-26T22:48:39.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.391+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a01a88b0 con 0x7fd2a0103e60 2024-04-26T22:48:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.392+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a006a110 con 0x7fd2a0103e60 2024-04-26T22:48:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.393+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd28c02a070 con 0x7fd2a0103e60 2024-04-26T22:48:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.394+0000 7fd29dffb700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd2900a3cd0 0x7fd2900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.394+0000 7fd2a4cea700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd2900a3cd0 0x7fd2900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.394+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fd28c0d1300 con 0x7fd2a0103e60 2024-04-26T22:48:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.395+0000 7fd2a4cea700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd2900a3cd0 0x7fd2900a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd2940045b0 tx=0x7fd29400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:39.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.399+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd28c013070 con 0x7fd2a0103e60 2024-04-26T22:48:39.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.609+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd2a002e490 con 0x7fd2a0103e60 2024-04-26T22:48:39.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.609+0000 7fd29dffb700 1 -- 172.21.15.1:0/374056674 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd28c09dc40 con 0x7fd2a0103e60 2024-04-26T22:48:39.610 INFO:teuthology.orchestra.run.smithi001.stdout:141733920829 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd2900a3cd0 msgr2=0x7fd2900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd2900a3cd0 0x7fd2900a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd2940045b0 tx=0x7fd29400b3c0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 msgr2=0x7fd2a019c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 0x7fd2a019c270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd28c000c00 tx=0x7fd28c0047b0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 shutdown_connections 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd2900a3cd0 0x7fd2900a6190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a01094e0 0x7fd2a01a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2a01034a0 0x7fd2a019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 --2- 172.21.15.1:0/374056674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2a0103e60 0x7fd2a019c270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:39.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 >> 172.21.15.1:0/374056674 conn(0x7fd2a007a590 msgr2=0x7fd2a010e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:39.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 shutdown_connections 2024-04-26T22:48:39.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:39.612+0000 7fd2a6f4e700 1 -- 172.21.15.1:0/374056674 wait complete. 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 -- 172.21.15.1:0/3731029858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 msgr2=0x7f8f50103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/3731029858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50103900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009990 tx=0x7f8f38009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 -- 172.21.15.1:0/3731029858 shutdown_connections 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/3731029858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f501094e0 0x7f8f501098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/3731029858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f50103e40 0x7f8f50110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/3731029858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50103900 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 -- 172.21.15.1:0/3731029858 >> 172.21.15.1:0/3731029858 conn(0x7f8f5007a4f0 msgr2=0x7f8f5007a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 -- 172.21.15.1:0/3731029858 shutdown_connections 2024-04-26T22:48:40.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.076+0000 7f8f55f8b700 1 -- 172.21.15.1:0/3731029858 wait complete. 2024-04-26T22:48:40.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 Processor -- start 2024-04-26T22:48:40.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 -- start start 2024-04-26T22:48:40.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f50103e40 0x7f8f50075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f501094e0 0x7f8f500761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8f501135f0 con 0x7f8f501034a0 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f50113470 con 0x7f8f50103e40 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f55f8b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8f50113770 con 0x7f8f501094e0 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f4f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f4f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36520/0 (socket says 172.21.15.1:36520) 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.077+0000 7f8f4f7fe700 1 -- 172.21.15.1:0/1517332720 learned_addr learned my addr 172.21.15.1:0/1517332720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f47fff700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f50103e40 0x7f8f50075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4ffff700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f501094e0 0x7f8f500761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4f7fe700 1 -- 172.21.15.1:0/1517332720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f501094e0 msgr2=0x7f8f500761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4f7fe700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f501094e0 0x7f8f500761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4f7fe700 1 -- 172.21.15.1:0/1517332720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f50103e40 msgr2=0x7f8f50075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4f7fe700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f50103e40 0x7f8f50075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4f7fe700 1 -- 172.21.15.1:0/1517332720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f501a8090 con 0x7f8f501034a0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4f7fe700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50075720 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000e9c0 tx=0x7f8f4000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4000cbd0 con 0x7f8f501034a0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f4000ee60 con 0x7f8f501034a0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f501a8380 con 0x7f8f501034a0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f400055c0 con 0x7f8f501034a0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.078+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f501a8840 con 0x7f8f501034a0 2024-04-26T22:48:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.079+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5006a110 con 0x7f8f501034a0 2024-04-26T22:48:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.081+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8f40004750 con 0x7f8f501034a0 2024-04-26T22:48:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.081+0000 7f8f4d7fa700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8f3c0a3cd0 0x7f8f3c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.081+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f8f400d0740 con 0x7f8f501034a0 2024-04-26T22:48:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.081+0000 7f8f47fff700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8f3c0a3cd0 0x7f8f3c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.082+0000 7f8f47fff700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8f3c0a3cd0 0x7f8f3c0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800b2e0 tx=0x7f8f38005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:40.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.086+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8f4009cff0 con 0x7f8f501034a0 2024-04-26T22:48:40.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:39 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/374056674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T22:48:40.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:39 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:48:40.303 INFO:teuthology.orchestra.run.smithi001.stdout:55834574925 2024-04-26T22:48:40.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.302+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8f5002e490 con 0x7f8f501034a0 2024-04-26T22:48:40.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.303+0000 7f8f4d7fa700 1 -- 172.21.15.1:0/1517332720 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f4009c8f0 con 0x7f8f501034a0 2024-04-26T22:48:40.304 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:39 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/374056674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T22:48:40.304 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:39 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.305+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8f3c0a3cd0 msgr2=0x7f8f3c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.305+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8f3c0a3cd0 0x7f8f3c0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800b2e0 tx=0x7f8f38005960 comp rx=0 tx=0).stop 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.305+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 msgr2=0x7f8f50075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.305+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50075720 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000e9c0 tx=0x7f8f4000ecd0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 shutdown_connections 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8f3c0a3cd0 0x7f8f3c0a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f501094e0 0x7f8f500761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f501034a0 0x7f8f50075720 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 --2- 172.21.15.1:0/1517332720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f50103e40 0x7f8f50075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 >> 172.21.15.1:0/1517332720 conn(0x7f8f5007a4f0 msgr2=0x7f8f501015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 shutdown_connections 2024-04-26T22:48:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.306+0000 7f8f55f8b700 1 -- 172.21.15.1:0/1517332720 wait complete. 2024-04-26T22:48:40.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:39 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/374056674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T22:48:40.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:39 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:48:40.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920829 for osd.4 2024-04-26T22:48:40.480 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 -- 172.21.15.1:0/1884983582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd784103e90 msgr2=0x7fd784110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 --2- 172.21.15.1:0/1884983582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd784103e90 0x7fd784110300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00b0d0 tx=0x7fd77c00b3e0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 -- 172.21.15.1:0/1884983582 shutdown_connections 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 --2- 172.21.15.1:0/1884983582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7841083b0 0x7fd784108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 --2- 172.21.15.1:0/1884983582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd784103e90 0x7fd784110300 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 --2- 172.21.15.1:0/1884983582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841034d0 0x7fd784103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 -- 172.21.15.1:0/1884983582 >> 172.21.15.1:0/1884983582 conn(0x7fd78407a580 msgr2=0x7fd78407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 -- 172.21.15.1:0/1884983582 shutdown_connections 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.711+0000 7fd78b45d700 1 -- 172.21.15.1:0/1884983582 wait complete. 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 Processor -- start 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 -- start start 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 0x7fd784075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd784103e90 0x7fd784075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841083b0 0x7fd7840761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd7841136a0 con 0x7fd7841034d0 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd784113520 con 0x7fd7841083b0 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd78b45d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd784113820 con 0x7fd784103e90 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd7891f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 0x7fd784075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd7891f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 0x7fd784075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36542/0 (socket says 172.21.15.1:36542) 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.712+0000 7fd7891f9700 1 -- 172.21.15.1:0/592331620 learned_addr learned my addr 172.21.15.1:0/592331620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7899fa700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841083b0 0x7fd7840761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:41.036 INFO:teuthology.orchestra.run.smithi001.stdout:77309411400 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7889f8700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd784103e90 0x7fd784075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7891f9700 1 -- 172.21.15.1:0/592331620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd784103e90 msgr2=0x7fd784075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7891f9700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd784103e90 0x7fd784075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7891f9700 1 -- 172.21.15.1:0/592331620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841083b0 msgr2=0x7fd7840761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7891f9700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841083b0 0x7fd7840761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7891f9700 1 -- 172.21.15.1:0/592331620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7841a8030 con 0x7fd7841034d0 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7899fa700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841083b0 0x7fd7840761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd7891f9700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 0x7fd784075720 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd78000b530 tx=0x7fd78000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780015020 con 0x7fd7841034d0 2024-04-26T22:48:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd780011ce0 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7841a8320 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780010dd0 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.713+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7841a8810 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.714+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78406a110 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.715+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd780018020 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.716+0000 7fd77a7fc700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd7700a3cd0 0x7fd7700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.716+0000 7fd7889f8700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd7700a3cd0 0x7fd7700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.717+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fd7800d00c0 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.717+0000 7fd7889f8700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd7700a3cd0 0x7fd7700a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd78407aaf0 tx=0x7fd77400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.720+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd78001f020 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.942+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd784076c00 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.943+0000 7fd77a7fc700 1 -- 172.21.15.1:0/592331620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd78009ca80 con 0x7fd7841034d0 2024-04-26T22:48:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd7700a3cd0 msgr2=0x7fd7700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd7700a3cd0 0x7fd7700a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd78407aaf0 tx=0x7fd77400b410 comp rx=0 tx=0).stop 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 msgr2=0x7fd784075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 0x7fd784075720 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd78000b530 tx=0x7fd78000b840 comp rx=0 tx=0).stop 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 shutdown_connections 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd7700a3cd0 0x7fd7700a6190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7841083b0 0x7fd7840761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7841034d0 0x7fd784075720 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 --2- 172.21.15.1:0/592331620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd784103e90 0x7fd784075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 >> 172.21.15.1:0/592331620 conn(0x7fd78407a580 msgr2=0x7fd784101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 shutdown_connections 2024-04-26T22:48:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:40.945+0000 7fd78b45d700 1 -- 172.21.15.1:0/592331620 wait complete. 2024-04-26T22:48:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:40 smithi038 ceph-mon[32433]: pgmap v322: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:40 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1517332720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T22:48:41.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:40 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/592331620' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T22:48:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:40 smithi186 ceph-mon[32774]: pgmap v322: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:40 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1517332720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T22:48:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:40 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/592331620' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T22:48:41.247 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:40 smithi001 ceph-mon[29435]: pgmap v322: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:41.247 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:40 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1517332720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T22:48:41.247 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:40 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/592331620' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T22:48:41.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574925 for osd.1 2024-04-26T22:48:41.904 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 -- 172.21.15.1:0/3004977791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30801094d0 msgr2=0x7f30801098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 --2- 172.21.15.1:0/3004977791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30801094d0 0x7f30801098b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3070009990 tx=0x7f3070009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 -- 172.21.15.1:0/3004977791 shutdown_connections 2024-04-26T22:48:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 --2- 172.21.15.1:0/3004977791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006c590 0x7f3080110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 --2- 172.21.15.1:0/3004977791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30801094d0 0x7f30801098b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 --2- 172.21.15.1:0/3004977791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f308006bbd0 0x7f308006c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.098+0000 7f3084dfe700 1 -- 172.21.15.1:0/3004977791 >> 172.21.15.1:0/3004977791 conn(0x7f30800793c0 msgr2=0x7f30800797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:42.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.099+0000 7f3084dfe700 1 -- 172.21.15.1:0/3004977791 shutdown_connections 2024-04-26T22:48:42.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.099+0000 7f3084dfe700 1 -- 172.21.15.1:0/3004977791 wait complete. 2024-04-26T22:48:42.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.099+0000 7f3084dfe700 1 Processor -- start 2024-04-26T22:48:42.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.099+0000 7f3084dfe700 1 -- start start 2024-04-26T22:48:42.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f3084dfe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006bbd0 0x7f308019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f3084dfe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 0x7f308019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f3084dfe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30801094d0 0x7f3080199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f3084dfe700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3080113600 con 0x7f308006c590 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f3084dfe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3080113480 con 0x7f308006bbd0 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f3084dfe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3080113780 con 0x7f30801094d0 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 0x7f308019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 0x7f308019fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60290/0 (socket says 172.21.15.1:60290) 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307dd9b700 1 -- 172.21.15.1:0/1338729250 learned_addr learned my addr 172.21.15.1:0/1338729250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307ed9d700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30801094d0 0x7f3080199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307e59c700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006bbd0 0x7f308019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307ed9d700 1 -- 172.21.15.1:0/1338729250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006bbd0 msgr2=0x7f308019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307ed9d700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006bbd0 0x7f308019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307ed9d700 1 -- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 msgr2=0x7f308019fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307ed9d700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 0x7f308019fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307ed9d700 1 -- 172.21.15.1:0/1338729250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308019a0e0 con 0x7f30801094d0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307e59c700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006bbd0 0x7f308019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.100+0000 7f307dd9b700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 0x7f308019fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.101+0000 7f307ed9d700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30801094d0 0x7f3080199980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3078007f10 tx=0x7f307800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.101+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3078004db0 con 0x7f30801094d0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.101+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308019a3d0 con 0x7f30801094d0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.101+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3078003a00 con 0x7f30801094d0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.101+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3078004580 con 0x7f30801094d0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.101+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30801a6640 con 0x7f30801094d0 2024-04-26T22:48:42.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.102+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308010ba40 con 0x7f30801094d0 2024-04-26T22:48:42.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.103+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f307801c020 con 0x7f30801094d0 2024-04-26T22:48:42.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.104+0000 7f306f7fe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30680a3cd0 0x7f30680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.104+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f30780cfe90 con 0x7f30801094d0 2024-04-26T22:48:42.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.104+0000 7f307e59c700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30680a3cd0 0x7f30680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.104+0000 7f307e59c700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30680a3cd0 0x7f30680a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f307000b5c0 tx=0x7f3070005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:42.106 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:42.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.107+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3078010070 con 0x7f30801094d0 2024-04-26T22:48:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 -- 172.21.15.1:0/468102746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08109490 msgr2=0x7f4d08109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/468102746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08109490 0x7f4d08109870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc009990 tx=0x7f4cfc009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 -- 172.21.15.1:0/468102746 shutdown_connections 2024-04-26T22:48:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/468102746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d0806c360 0x7f4d081102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/468102746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08109490 0x7f4d08109870 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/468102746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d0806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 -- 172.21.15.1:0/468102746 >> 172.21.15.1:0/468102746 conn(0x7f4d08079290 msgr2=0x7f4d080796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 -- 172.21.15.1:0/468102746 shutdown_connections 2024-04-26T22:48:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 -- 172.21.15.1:0/468102746 wait complete. 2024-04-26T22:48:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 Processor -- start 2024-04-26T22:48:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.283+0000 7f4d0f20a700 1 -- start start 2024-04-26T22:48:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0f20a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d081023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0f20a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d0806c360 0x7f4d08102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0f20a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 0x7f4d08102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0f20a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4d081135a0 con 0x7f4d0806c360 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0f20a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4d08113420 con 0x7f4d0806b9c0 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0f20a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d08113720 con 0x7f4d08109490 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d081023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0d7a7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 0x7f4d08102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d081023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41364/0 (socket says 172.21.15.1:41364) 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0d7a7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 0x7f4d08102e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43924/0 (socket says 172.21.15.1:43924) 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 -- 172.21.15.1:0/3555495440 learned_addr learned my addr 172.21.15.1:0/3555495440 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d07fff700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d0806c360 0x7f4d08102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 -- 172.21.15.1:0/3555495440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 msgr2=0x7f4d08102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 0x7f4d08102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 -- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d0806c360 msgr2=0x7f4d08102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d0806c360 0x7f4d08102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 -- 172.21.15.1:0/3555495440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d081a8040 con 0x7f4d0806b9c0 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0d7a7700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 0x7f4d08102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d0cfa6700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d081023e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc000c00 tx=0x7f4cfc00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.284+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc027070 con 0x7f4d0806b9c0 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.285+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cfc022470 con 0x7f4d0806b9c0 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.285+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d081a82d0 con 0x7f4d0806b9c0 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.285+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc017650 con 0x7f4d0806b9c0 2024-04-26T22:48:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.285+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d081a8780 con 0x7f4d0806b9c0 2024-04-26T22:48:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.286+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d0806a110 con 0x7f4d0806b9c0 2024-04-26T22:48:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.287+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4cfc02a070 con 0x7f4d0806b9c0 2024-04-26T22:48:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.287+0000 7f4d05ffb700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4cf80a3d50 0x7f4cf80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.288+0000 7f4d07fff700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4cf80a3d50 0x7f4cf80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.288+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f4cfc0d13c0 con 0x7f4d0806b9c0 2024-04-26T22:48:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.288+0000 7f4d07fff700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4cf80a3d50 0x7f4cf80a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d08079940 tx=0x7f4cf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.291+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4cfc013070 con 0x7f4d0806b9c0 2024-04-26T22:48:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.329+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f308010b420 con 0x7f30801094d0 2024-04-26T22:48:42.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.330+0000 7f306f7fe700 1 -- 172.21.15.1:0/1338729250 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f307809c740 con 0x7f30801094d0 2024-04-26T22:48:42.330 INFO:teuthology.orchestra.run.smithi001.stdout:253403070506 2024-04-26T22:48:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30680a3cd0 msgr2=0x7f30680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30680a3cd0 0x7f30680a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f307000b5c0 tx=0x7f3070005960 comp rx=0 tx=0).stop 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30801094d0 msgr2=0x7f3080199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30801094d0 0x7f3080199980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3078007f10 tx=0x7f307800d3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 shutdown_connections 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30680a3cd0 0x7f30680a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30801094d0 0x7f3080199980 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f308006bbd0 0x7f308019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 --2- 172.21.15.1:0/1338729250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f308006c590 0x7f308019fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 >> 172.21.15.1:0/1338729250 conn(0x7f30800793c0 msgr2=0x7f30801003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 shutdown_connections 2024-04-26T22:48:42.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.332+0000 7f3084dfe700 1 -- 172.21.15.1:0/1338729250 wait complete. 2024-04-26T22:48:42.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.506+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4d0802e490 con 0x7f4d0806b9c0 2024-04-26T22:48:42.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.507+0000 7f4d05ffb700 1 -- 172.21.15.1:0/3555495440 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4cfc09dca0 con 0x7f4d0806b9c0 2024-04-26T22:48:42.507 INFO:teuthology.orchestra.run.smithi001.stdout:197568495668 2024-04-26T22:48:42.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4cf80a3d50 msgr2=0x7f4cf80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4cf80a3d50 0x7f4cf80a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d08079940 tx=0x7f4cf4009260 comp rx=0 tx=0).stop 2024-04-26T22:48:42.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 msgr2=0x7f4d081023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:42.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d081023e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc000c00 tx=0x7f4cfc00bfd0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 shutdown_connections 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4cf80a3d50 0x7f4cf80a6210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d08109490 0x7f4d08102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d0806b9c0 0x7f4d081023e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 --2- 172.21.15.1:0/3555495440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d0806c360 0x7f4d08102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 >> 172.21.15.1:0/3555495440 conn(0x7f4d08079290 msgr2=0x7f4d08100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 shutdown_connections 2024-04-26T22:48:42.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:42.509+0000 7f4d0f20a700 1 -- 172.21.15.1:0/3555495440 wait complete. 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: pgmap v323: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1338729250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T22:48:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3555495440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: pgmap v323: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1338729250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T22:48:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3555495440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T22:48:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:43.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: pgmap v323: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T22:48:43.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:43.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:43.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1338729250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T22:48:43.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3555495440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T22:48:44.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495665 got 197568495668 for osd.6 2024-04-26T22:48:44.442 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:44.873 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411395 got 77309411400 for osd.2 2024-04-26T22:48:44.873 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:45.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:44 smithi001 ceph-mon[29435]: pgmap v324: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T22:48:45.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:44 smithi038 ceph-mon[32433]: pgmap v324: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T22:48:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:44 smithi186 ceph-mon[32774]: pgmap v324: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T22:48:45.288 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:45.288 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:45.292 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:45.315 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070501 got 253403070506 for osd.8 2024-04-26T22:48:45.316 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 -- 172.21.15.1:0/2569150748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 msgr2=0x7fb798103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 --2- 172.21.15.1:0/2569150748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb798103940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0099c0 tx=0x7fb78c009cd0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 -- 172.21.15.1:0/2569150748 shutdown_connections 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 --2- 172.21.15.1:0/2569150748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798103e80 0x7fb798110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 --2- 172.21.15.1:0/2569150748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb798109560 0x7fb798109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 --2- 172.21.15.1:0/2569150748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb798103940 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 -- 172.21.15.1:0/2569150748 >> 172.21.15.1:0/2569150748 conn(0x7fb79807a5d0 msgr2=0x7fb79807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 -- 172.21.15.1:0/2569150748 shutdown_connections 2024-04-26T22:48:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.200+0000 7fb79ca48700 1 -- 172.21.15.1:0/2569150748 wait complete. 2024-04-26T22:48:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 Processor -- start 2024-04-26T22:48:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 -- start start 2024-04-26T22:48:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb79819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798103e80 0x7fb79819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb798109560 0x7fb798199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb798113670 con 0x7fb7981034c0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb7981134f0 con 0x7fb798109560 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79ca48700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb7981137f0 con 0x7fb798103e80 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb79819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb79819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60320/0 (socket says 172.21.15.1:60320) 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb79659c700 1 -- 172.21.15.1:0/3099537566 learned_addr learned my addr 172.21.15.1:0/3099537566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.201+0000 7fb795d9b700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798103e80 0x7fb79819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb796d9d700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb798109560 0x7fb798199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79659c700 1 -- 172.21.15.1:0/3099537566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798103e80 msgr2=0x7fb79819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79659c700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798103e80 0x7fb79819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79659c700 1 -- 172.21.15.1:0/3099537566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb798109560 msgr2=0x7fb798199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79659c700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb798109560 0x7fb798199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79659c700 1 -- 172.21.15.1:0/3099537566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79819a090 con 0x7fb7981034c0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79659c700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb79819f720 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb78800c9b0 tx=0x7fb78800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788007b00 con 0x7fb7981034c0 2024-04-26T22:48:46.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb788007ca0 con 0x7fb7981034c0 2024-04-26T22:48:46.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb79819a380 con 0x7fb7981034c0 2024-04-26T22:48:46.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.202+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788018660 con 0x7fb7981034c0 2024-04-26T22:48:46.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.203+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7981a65f0 con 0x7fb7981034c0 2024-04-26T22:48:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.204+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79806a110 con 0x7fb7981034c0 2024-04-26T22:48:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.204+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb78800f450 con 0x7fb7981034c0 2024-04-26T22:48:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.205+0000 7fb7877fe700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb7800a3d50 0x7fb7800a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.205+0000 7fb795d9b700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb7800a3d50 0x7fb7800a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.205+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fb7880d0440 con 0x7fb7981034c0 2024-04-26T22:48:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.206+0000 7fb795d9b700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb7800a3d50 0x7fb7800a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009990 tx=0x7fb78c00b790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:46.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.210+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb788014030 con 0x7fb7981034c0 2024-04-26T22:48:46.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.432+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb79819ae10 con 0x7fb7981034c0 2024-04-26T22:48:46.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.432+0000 7fb7877fe700 1 -- 172.21.15.1:0/3099537566 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb78809ccf0 con 0x7fb7981034c0 2024-04-26T22:48:46.433 INFO:teuthology.orchestra.run.smithi001.stdout:163208757307 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb7800a3d50 msgr2=0x7fb7800a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb7800a3d50 0x7fb7800a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009990 tx=0x7fb78c00b790 comp rx=0 tx=0).stop 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 msgr2=0x7fb79819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb79819f720 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb78800c9b0 tx=0x7fb78800ccc0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 shutdown_connections 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fb7800a3d50 0x7fb7800a6210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb798109560 0x7fb798199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7981034c0 0x7fb79819f720 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 --2- 172.21.15.1:0/3099537566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798103e80 0x7fb79819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 >> 172.21.15.1:0/3099537566 conn(0x7fb79807a5d0 msgr2=0x7fb798102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 shutdown_connections 2024-04-26T22:48:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:46.435+0000 7fb79ca48700 1 -- 172.21.15.1:0/3099537566 wait complete. 2024-04-26T22:48:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:46 smithi001 ceph-mon[29435]: pgmap v325: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3099537566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T22:48:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:46 smithi038 ceph-mon[32433]: pgmap v325: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3099537566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T22:48:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:46 smithi186 ceph-mon[32774]: pgmap v325: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3099537566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T22:48:47.273 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757307 for osd.5 2024-04-26T22:48:47.274 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 -- 172.21.15.1:0/1243234769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406baf0 msgr2=0x7f260406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 --2- 172.21.15.1:0/1243234769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406baf0 0x7f260406bf50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009990 tx=0x7f25f4009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 -- 172.21.15.1:0/1243234769 shutdown_connections 2024-04-26T22:48:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 --2- 172.21.15.1:0/1243234769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f260406c490 0x7f2604110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 --2- 172.21.15.1:0/1243234769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26041094c0 0x7f26041098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 --2- 172.21.15.1:0/1243234769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406baf0 0x7f260406bf50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 -- 172.21.15.1:0/1243234769 >> 172.21.15.1:0/1243234769 conn(0x7f26040792d0 msgr2=0x7f26040796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 -- 172.21.15.1:0/1243234769 shutdown_connections 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 -- 172.21.15.1:0/1243234769 wait complete. 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.454+0000 7f260b817700 1 Processor -- start 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 -- start start 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f260406baf0 0x7f26041027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 0x7f2604102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26041094c0 0x7f2604103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f26041135e0 con 0x7f260406c490 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2604113460 con 0x7f260406baf0 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f260b817700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2604113760 con 0x7f26041094c0 2024-04-26T22:48:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 0x7f2604102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 0x7f2604102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60340/0 (socket says 172.21.15.1:60340) 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 -- 172.21.15.1:0/1991784090 learned_addr learned my addr 172.21.15.1:0/1991784090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2609db4700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26041094c0 0x7f2604103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f26095b3700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f260406baf0 0x7f26041027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 -- 172.21.15.1:0/1991784090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26041094c0 msgr2=0x7f2604103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26041094c0 0x7f2604103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 -- 172.21.15.1:0/1991784090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f260406baf0 msgr2=0x7f26041027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f260406baf0 0x7f26041027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 -- 172.21.15.1:0/1991784090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26041a8080 con 0x7f260406c490 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f2608db2700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 0x7f2604102d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25f4000c00 tx=0x7f25f40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.455+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4027070 con 0x7f260406c490 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.456+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26041a8310 con 0x7f260406c490 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.456+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25f4003fa0 con 0x7f260406c490 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.456+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4022db0 con 0x7f260406c490 2024-04-26T22:48:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.456+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26041a8760 con 0x7f260406c490 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.456+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2604104b70 con 0x7f260406c490 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.458+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f25f402a070 con 0x7f260406c490 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.458+0000 7f25fa7fc700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f25f00a3cd0 0x7f25f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.458+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f25f40d1650 con 0x7f260406c490 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.458+0000 7f26095b3700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f25f00a3cd0 0x7f25f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.459+0000 7f26095b3700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f25f00a3cd0 0x7f25f00a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2600005950 tx=0x7f26000058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.462+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f25f4013070 con 0x7f260406c490 2024-04-26T22:48:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.682+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f260402e490 con 0x7f260406c490 2024-04-26T22:48:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.682+0000 7f25fa7fc700 1 -- 172.21.15.1:0/1991784090 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f25f409df90 con 0x7f260406c490 2024-04-26T22:48:47.683 INFO:teuthology.orchestra.run.smithi001.stdout:309237645344 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.684+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f25f00a3cd0 msgr2=0x7f25f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.684+0000 7f260b817700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f25f00a3cd0 0x7f25f00a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2600005950 tx=0x7f26000058e0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 msgr2=0x7f2604102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 0x7f2604102d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25f4000c00 tx=0x7f25f40047b0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 shutdown_connections 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f25f00a3cd0 0x7f25f00a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26041094c0 0x7f2604103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f260406baf0 0x7f26041027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 --2- 172.21.15.1:0/1991784090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260406c490 0x7f2604102d10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 >> 172.21.15.1:0/1991784090 conn(0x7f26040792d0 msgr2=0x7f2604100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:47.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 shutdown_connections 2024-04-26T22:48:47.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:47.685+0000 7f260b817700 1 -- 172.21.15.1:0/1991784090 wait complete. 2024-04-26T22:48:48.098 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:47 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1991784090' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T22:48:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:47 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1991784090' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T22:48:48.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:47 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1991784090' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T22:48:49.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:48 smithi001 ceph-mon[29435]: pgmap v326: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:49.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:48 smithi038 ceph-mon[32433]: pgmap v326: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:48 smithi186 ceph-mon[32774]: pgmap v326: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:49.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645338 got 309237645344 for osd.10 2024-04-26T22:48:49.805 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 -- 172.21.15.1:0/4212697838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 msgr2=0x7f0f0c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 --2- 172.21.15.1:0/4212697838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c108770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009990 tx=0x7f0efc009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 -- 172.21.15.1:0/4212697838 shutdown_connections 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 --2- 172.21.15.1:0/4212697838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f0c102ce0 0x7f0f0c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 --2- 172.21.15.1:0/4212697838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c108770 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 --2- 172.21.15.1:0/4212697838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f0c102320 0x7f0f0c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 -- 172.21.15.1:0/4212697838 >> 172.21.15.1:0/4212697838 conn(0x7f0f0c0792d0 msgr2=0x7f0f0c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 -- 172.21.15.1:0/4212697838 shutdown_connections 2024-04-26T22:48:50.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.595+0000 7f0f108e3700 1 -- 172.21.15.1:0/4212697838 wait complete. 2024-04-26T22:48:50.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.596+0000 7f0f108e3700 1 Processor -- start 2024-04-26T22:48:50.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.596+0000 7f0f108e3700 1 -- start start 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.596+0000 7f0f108e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f0c102320 0x7f0f0c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.596+0000 7f0f108e3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f0c102ce0 0x7f0f0c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f108e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f108e3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1124b0 con 0x7f0f0c108390 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f108e3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c112330 con 0x7f0f0c102320 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f108e3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c112630 con 0x7f0f0c102ce0 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60358/0 (socket says 172.21.15.1:60358) 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 -- 172.21.15.1:0/364832995 learned_addr learned my addr 172.21.15.1:0/364832995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f03fff700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f0c102ce0 0x7f0f0c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0a59c700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f0c102320 0x7f0f0c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 -- 172.21.15.1:0/364832995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f0c102ce0 msgr2=0x7f0f0c19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f0c102ce0 0x7f0f0c19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 -- 172.21.15.1:0/364832995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f0c102320 msgr2=0x7f0f0c19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f0c102320 0x7f0f0c19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 -- 172.21.15.1:0/364832995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c199ff0 con 0x7f0f0c108390 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f0ad9d700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c199890 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400ef30 tx=0x7f0f0400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f040092e0 con 0x7f0f0c108390 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f04004bb0 con 0x7f0f0c108390 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c19a280 con 0x7f0f0c108390 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04004020 con 0x7f0f0c108390 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.597+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c1a65a0 con 0x7f0f0c108390 2024-04-26T22:48:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.598+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c06a110 con 0x7f0f0c108390 2024-04-26T22:48:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.600+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f0f0401b030 con 0x7f0f0c108390 2024-04-26T22:48:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.600+0000 7f0f037fe700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0ef80a3cd0 0x7f0ef80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.600+0000 7f0f0a59c700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0ef80a3cd0 0x7f0ef80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.600+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f0f040d0920 con 0x7f0f0c108390 2024-04-26T22:48:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.601+0000 7f0f0a59c700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0ef80a3cd0 0x7f0ef80a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00b2e0 tx=0x7f0efc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:50.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.604+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0f04022020 con 0x7f0f0c108390 2024-04-26T22:48:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.814+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f0f0c1a6770 con 0x7f0f0c108390 2024-04-26T22:48:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.814+0000 7f0f037fe700 1 -- 172.21.15.1:0/364832995 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0f04010040 con 0x7f0f0c108390 2024-04-26T22:48:50.814 INFO:teuthology.orchestra.run.smithi001.stdout:219043332145 2024-04-26T22:48:50.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0ef80a3cd0 msgr2=0x7f0ef80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0ef80a3cd0 0x7f0ef80a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00b2e0 tx=0x7f0efc01a040 comp rx=0 tx=0).stop 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 msgr2=0x7f0f0c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c199890 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400ef30 tx=0x7f0f0400c3a0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 shutdown_connections 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0ef80a3cd0 0x7f0ef80a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f0c108390 0x7f0f0c199890 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f0c102320 0x7f0f0c19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 --2- 172.21.15.1:0/364832995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f0c102ce0 0x7f0f0c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 >> 172.21.15.1:0/364832995 conn(0x7f0f0c0792d0 msgr2=0x7f0f0c101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 shutdown_connections 2024-04-26T22:48:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:50.816+0000 7f0f108e3700 1 -- 172.21.15.1:0/364832995 wait complete. 2024-04-26T22:48:51.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:50 smithi001 ceph-mon[29435]: pgmap v327: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:51.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:50 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/364832995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T22:48:51.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:50 smithi038 ceph-mon[32433]: pgmap v327: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:51.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:50 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/364832995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T22:48:51.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:50 smithi186 ceph-mon[32774]: pgmap v327: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:51.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:50 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/364832995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T22:48:52.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332145 for osd.7 2024-04-26T22:48:52.762 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:53.048 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:52 smithi001 ceph-mon[29435]: pgmap v328: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:48:53.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:52 smithi038 ceph-mon[32433]: pgmap v328: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:48:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:52 smithi186 ceph-mon[32774]: pgmap v328: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:48:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 -- 172.21.15.1:0/127297375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde381083c0 msgr2=0x7fde381087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/127297375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde381083c0 0x7fde381087a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fde280098f0 tx=0x7fde28009c00 comp rx=0 tx=0).stop 2024-04-26T22:48:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 -- 172.21.15.1:0/127297375 shutdown_connections 2024-04-26T22:48:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/127297375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102d10 0x7fde3810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/127297375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde381083c0 0x7fde381087a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/127297375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde38102350 0x7fde381027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 -- 172.21.15.1:0/127297375 >> 172.21.15.1:0/127297375 conn(0x7fde38079360 msgr2=0x7fde38079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 -- 172.21.15.1:0/127297375 shutdown_connections 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.184+0000 7fde3e9aa700 1 -- 172.21.15.1:0/127297375 wait complete. 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.185+0000 7fde3e9aa700 1 Processor -- start 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.185+0000 7fde3e9aa700 1 -- start start 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3e9aa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 0x7fde381092a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3e9aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde38102d10 0x7fde381097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3e9aa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde381083c0 0x7fde38109d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3e9aa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fde38112510 con 0x7fde38102d10 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3e9aa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fde38112390 con 0x7fde381083c0 2024-04-26T22:48:53.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3e9aa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fde38112690 con 0x7fde38102350 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 0x7fde381092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 0x7fde381092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51998/0 (socket says 172.21.15.1:51998) 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 -- 172.21.15.1:0/3211355094 learned_addr learned my addr 172.21.15.1:0/3211355094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde3cf47700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde381083c0 0x7fde38109d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stdout:287762808871 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 -- 172.21.15.1:0/3211355094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde381083c0 msgr2=0x7fde38109d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde381083c0 0x7fde38109d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 -- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde38102d10 msgr2=0x7fde381097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde377fe700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde38102d10 0x7fde381097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde38102d10 0x7fde381097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde37fff700 1 -- 172.21.15.1:0/3211355094 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde381a81d0 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.186+0000 7fde377fe700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde38102d10 0x7fde381097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.187+0000 7fde37fff700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 0x7fde381092a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde2800f990 tx=0x7fde2800f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.187+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28027070 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.187+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde280055f0 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.187+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28004930 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.187+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde381a8460 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.187+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde381a8800 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.189+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde38103a70 con 0x7fde38102350 2024-04-26T22:48:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.190+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fde2802a070 con 0x7fde38102350 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.191+0000 7fde357fa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fde200a3cd0 0x7fde200a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.191+0000 7fde377fe700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fde200a3cd0 0x7fde200a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.191+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fde280d0f20 con 0x7fde38102350 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.192+0000 7fde377fe700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fde200a3cd0 0x7fde200a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde38079960 tx=0x7fde2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.196+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fde2805f060 con 0x7fde38102350 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.408+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fde3810a780 con 0x7fde38102350 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.408+0000 7fde357fa700 1 -- 172.21.15.1:0/3211355094 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fde28013030 con 0x7fde38102350 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fde200a3cd0 msgr2=0x7fde200a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fde200a3cd0 0x7fde200a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde38079960 tx=0x7fde2c009260 comp rx=0 tx=0).stop 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 msgr2=0x7fde381092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 0x7fde381092a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde2800f990 tx=0x7fde2800f9c0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 shutdown_connections 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fde200a3cd0 0x7fde200a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde38102350 0x7fde381092a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde381083c0 0x7fde38109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 --2- 172.21.15.1:0/3211355094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde38102d10 0x7fde381097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 >> 172.21.15.1:0/3211355094 conn(0x7fde38079360 msgr2=0x7fde38100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 shutdown_connections 2024-04-26T22:48:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:53.410+0000 7fde3e9aa700 1 -- 172.21.15.1:0/3211355094 wait complete. 2024-04-26T22:48:54.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:53 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3211355094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T22:48:54.248 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:53 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3211355094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T22:48:54.275 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808871 for osd.9 2024-04-26T22:48:54.276 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:54.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:53 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3211355094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T22:48:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 -- 172.21.15.1:0/3224913637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 msgr2=0x7eff9c107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 --2- 172.21.15.1:0/3224913637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c107780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00b0d0 tx=0x7eff8c00b3e0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 -- 172.21.15.1:0/3224913637 shutdown_connections 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 --2- 172.21.15.1:0/3224913637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff9c103030 0x7eff9c10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 --2- 172.21.15.1:0/3224913637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c107780 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 --2- 172.21.15.1:0/3224913637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 0x7eff9c102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 -- 172.21.15.1:0/3224913637 >> 172.21.15.1:0/3224913637 conn(0x7eff9c078190 msgr2=0x7eff9c0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 -- 172.21.15.1:0/3224913637 shutdown_connections 2024-04-26T22:48:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 -- 172.21.15.1:0/3224913637 wait complete. 2024-04-26T22:48:54.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.722+0000 7effa1d54700 1 Processor -- start 2024-04-26T22:48:54.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 -- start start 2024-04-26T22:48:54.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 0x7eff9c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff9c103030 0x7eff9c19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c1a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c1125d0 con 0x7eff9c1073a0 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c112450 con 0x7eff9c102670 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7effa1d54700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c112750 con 0x7eff9c103030 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c1a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 0x7eff9c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c1a59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52720/0 (socket says 172.21.15.1:52720) 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 0x7eff9c19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57330/0 (socket says 172.21.15.1:57330) 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 -- 172.21.15.1:0/3059912695 learned_addr learned my addr 172.21.15.1:0/3059912695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9affd700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff9c103030 0x7eff9c19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 -- 172.21.15.1:0/3059912695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff9c103030 msgr2=0x7eff9c19ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff9c103030 0x7eff9c19ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 -- 172.21.15.1:0/3059912695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 msgr2=0x7eff9c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:54.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 0x7eff9c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.723+0000 7eff9bfff700 1 -- 172.21.15.1:0/3059912695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c1a5ef0 con 0x7eff9c1073a0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.724+0000 7eff9bfff700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c1a59b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff9400b550 tx=0x7eff9400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.724+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94015020 con 0x7eff9c1073a0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.724+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff94011ce0 con 0x7eff9c1073a0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.724+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9c1a6120 con 0x7eff9c1073a0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.724+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94010c90 con 0x7eff9c1073a0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.724+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9c1a6610 con 0x7eff9c1073a0 2024-04-26T22:48:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.725+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9c103a90 con 0x7eff9c1073a0 2024-04-26T22:48:54.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.726+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7eff94018020 con 0x7eff9c1073a0 2024-04-26T22:48:54.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.727+0000 7eff98ff9700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7eff880a3cd0 0x7eff880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:54.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.727+0000 7eff9b7fe700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7eff880a3cd0 0x7eff880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:54.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.727+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7eff940cfaa0 con 0x7eff9c1073a0 2024-04-26T22:48:54.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.727+0000 7eff9b7fe700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7eff880a3cd0 0x7eff880a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009fd0 tx=0x7eff8c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:54.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.732+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7eff9401f020 con 0x7eff9c1073a0 2024-04-26T22:48:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.952+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7eff9c02e650 con 0x7eff9c1073a0 2024-04-26T22:48:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.952+0000 7eff98ff9700 1 -- 172.21.15.1:0/3059912695 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff9409c460 con 0x7eff9c1073a0 2024-04-26T22:48:54.954 INFO:teuthology.orchestra.run.smithi001.stdout:34359738451 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7eff880a3cd0 msgr2=0x7eff880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7eff880a3cd0 0x7eff880a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009fd0 tx=0x7eff8c00b040 comp rx=0 tx=0).stop 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 msgr2=0x7eff9c1a59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c1a59b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff9400b550 tx=0x7eff9400b860 comp rx=0 tx=0).stop 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 shutdown_connections 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7eff880a3cd0 0x7eff880a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff9c1073a0 0x7eff9c1a59b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff9c102670 0x7eff9c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 --2- 172.21.15.1:0/3059912695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff9c103030 0x7eff9c19ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 >> 172.21.15.1:0/3059912695 conn(0x7eff9c078190 msgr2=0x7eff9c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 shutdown_connections 2024-04-26T22:48:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:54.955+0000 7effa1d54700 1 -- 172.21.15.1:0/3059912695 wait complete. 2024-04-26T22:48:55.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:54 smithi038 ceph-mon[32433]: pgmap v329: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:55.190 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:54 smithi001 ceph-mon[29435]: pgmap v329: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:55.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:54 smithi186 ceph-mon[32774]: pgmap v329: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:48:55.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1726397865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44102320 msgr2=0x7fbd441027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1726397865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44102320 0x7fbd441027a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009990 tx=0x7fbd38009ca0 comp rx=0 tx=0).stop 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1726397865 shutdown_connections 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1726397865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44108390 0x7fbd44108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1726397865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44102ce0 0x7fbd4410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1726397865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44102320 0x7fbd441027a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1726397865 >> 172.21.15.1:0/1726397865 conn(0x7fbd440792d0 msgr2=0x7fbd440796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.893+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1726397865 shutdown_connections 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.894+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1726397865 wait complete. 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.894+0000 7fbd4ac2f700 1 Processor -- start 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.894+0000 7fbd4ac2f700 1 -- start start 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd4ac2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44102320 0x7fbd4419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd4ac2f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44102ce0 0x7fbd4419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd4ac2f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 0x7fbd441a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd4ac2f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbd441124d0 con 0x7fbd44108390 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd4ac2f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44112350 con 0x7fbd44102320 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd4ac2f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44112650 con 0x7fbd44102ce0 2024-04-26T22:48:55.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 0x7fbd441a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 0x7fbd441a59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52740/0 (socket says 172.21.15.1:52740) 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 -- 172.21.15.1:0/1044773438 learned_addr learned my addr 172.21.15.1:0/1044773438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd43fff700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44102ce0 0x7fbd4419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd489cb700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44102320 0x7fbd4419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 -- 172.21.15.1:0/1044773438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44102ce0 msgr2=0x7fbd4419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44102ce0 0x7fbd4419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 -- 172.21.15.1:0/1044773438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44102320 msgr2=0x7fbd4419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44102320 0x7fbd4419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 -- 172.21.15.1:0/1044773438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd441a5f30 con 0x7fbd44108390 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd491cc700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 0x7fbd441a59f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c007ee0 tx=0x7fbd3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.895+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c00dbd0 con 0x7fbd44108390 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.896+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd3c00dd70 con 0x7fbd44108390 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.896+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c005990 con 0x7fbd44108390 2024-04-26T22:48:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.896+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd441a6160 con 0x7fbd44108390 2024-04-26T22:48:55.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.896+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd441a6620 con 0x7fbd44108390 2024-04-26T22:48:55.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.897+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd4406a110 con 0x7fbd44108390 2024-04-26T22:48:55.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.898+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbd3c01a020 con 0x7fbd44108390 2024-04-26T22:48:55.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.899+0000 7fbd41ffb700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbd2c0a3d50 0x7fbd2c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:55.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.899+0000 7fbd489cb700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbd2c0a3d50 0x7fbd2c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:55.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.899+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fbd3c0cf750 con 0x7fbd44108390 2024-04-26T22:48:55.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.899+0000 7fbd489cb700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbd2c0a3d50 0x7fbd2c0a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd34005fd0 tx=0x7fbd34005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:55.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:55.903+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbd3c010070 con 0x7fbd44108390 2024-04-26T22:48:56.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:55 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3059912695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T22:48:56.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.113+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbd4402e5b0 con 0x7fbd44108390 2024-04-26T22:48:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.114+0000 7fbd41ffb700 1 -- 172.21.15.1:0/1044773438 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbd3c0112b0 con 0x7fbd44108390 2024-04-26T22:48:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:107374182471 2024-04-26T22:48:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbd2c0a3d50 msgr2=0x7fbd2c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbd2c0a3d50 0x7fbd2c0a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd34005fd0 tx=0x7fbd34005d90 comp rx=0 tx=0).stop 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 msgr2=0x7fbd441a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 0x7fbd441a59f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c007ee0 tx=0x7fbd3c00d3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 shutdown_connections 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbd2c0a3d50 0x7fbd2c0a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44108390 0x7fbd441a59f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44102320 0x7fbd4419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 --2- 172.21.15.1:0/1044773438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44102ce0 0x7fbd4419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 >> 172.21.15.1:0/1044773438 conn(0x7fbd440792d0 msgr2=0x7fbd44100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 shutdown_connections 2024-04-26T22:48:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.116+0000 7fbd4ac2f700 1 -- 172.21.15.1:0/1044773438 wait complete. 2024-04-26T22:48:56.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:55 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3059912695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T22:48:56.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:55 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3059912695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T22:48:56.598 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738451 for osd.0 2024-04-26T22:48:56.598 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:56.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 -- 172.21.15.1:0/4054149521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c601094c0 msgr2=0x7f2c601098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:56.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 --2- 172.21.15.1:0/4054149521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c601094c0 0x7f2c601098a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000b0a0 tx=0x7f2c5000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 -- 172.21.15.1:0/4054149521 shutdown_connections 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 --2- 172.21.15.1:0/4054149521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c60103e30 0x7f2c60110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 --2- 172.21.15.1:0/4054149521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c601094c0 0x7f2c601098a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 --2- 172.21.15.1:0/4054149521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 0x7f2c601038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.819+0000 7f2c6526e700 1 -- 172.21.15.1:0/4054149521 >> 172.21.15.1:0/4054149521 conn(0x7f2c6007a580 msgr2=0x7f2c6007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.820+0000 7f2c6526e700 1 -- 172.21.15.1:0/4054149521 shutdown_connections 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.820+0000 7f2c6526e700 1 -- 172.21.15.1:0/4054149521 wait complete. 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.820+0000 7f2c6526e700 1 Processor -- start 2024-04-26T22:48:56.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.820+0000 7f2c6526e700 1 -- start start 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c6526e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 0x7f2c6019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c6526e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 0x7f2c6019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c6526e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c601094c0 0x7f2c60199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c6526e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2c60113640 con 0x7f2c60103e30 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c6526e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601134c0 con 0x7f2c601094c0 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c6526e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601137c0 con 0x7f2c60103470 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 0x7f2c6019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 0x7f2c6019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 0x7f2c6019fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52746/0 (socket says 172.21.15.1:52746) 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 -- 172.21.15.1:0/3764125181 learned_addr learned my addr 172.21.15.1:0/3764125181 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5f7fe700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c601094c0 0x7f2c60199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 -- 172.21.15.1:0/3764125181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 msgr2=0x7f2c6019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 0x7f2c6019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 -- 172.21.15.1:0/3764125181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c601094c0 msgr2=0x7f2c60199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c601094c0 0x7f2c60199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 -- 172.21.15.1:0/3764125181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6019a0e0 con 0x7f2c60103e30 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5effd700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 0x7f2c6019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.821+0000 7f2c5e7fc700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 0x7f2c6019fd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400c920 tx=0x7f2c5400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.822+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c54004a30 con 0x7f2c60103e30 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.822+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c540054d0 con 0x7f2c60103e30 2024-04-26T22:48:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.822+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6019a3d0 con 0x7f2c60103e30 2024-04-26T22:48:56.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.822+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c601a6690 con 0x7f2c60103e30 2024-04-26T22:48:56.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.822+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c54007c70 con 0x7f2c60103e30 2024-04-26T22:48:56.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.823+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4c005330 con 0x7f2c60103e30 2024-04-26T22:48:56.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.824+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2c54005020 con 0x7f2c60103e30 2024-04-26T22:48:56.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.825+0000 7f2c47fff700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2c480a3cd0 0x7f2c480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:48:56.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.825+0000 7f2c5effd700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2c480a3cd0 0x7f2c480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:48:56.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.825+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f2c540d0370 con 0x7f2c60103e30 2024-04-26T22:48:56.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.825+0000 7f2c5effd700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2c480a3cd0 0x7f2c480a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009e60 tx=0x7f2c5000b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:48:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:56.829+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c54014030 con 0x7f2c60103e30 2024-04-26T22:48:57.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.047+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f2c4c0058f0 con 0x7f2c60103e30 2024-04-26T22:48:57.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.047+0000 7f2c47fff700 1 -- 172.21.15.1:0/3764125181 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f2c5409cbc0 con 0x7f2c60103e30 2024-04-26T22:48:57.048 INFO:teuthology.orchestra.run.smithi001.stdout:343597383709 2024-04-26T22:48:57.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2c480a3cd0 msgr2=0x7f2c480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:57.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2c480a3cd0 0x7f2c480a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009e60 tx=0x7f2c5000b7b0 comp rx=0 tx=0).stop 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 msgr2=0x7f2c6019fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 0x7f2c6019fd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400c920 tx=0x7f2c5400cc30 comp rx=0 tx=0).stop 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 shutdown_connections 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2c480a3cd0 0x7f2c480a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c60103470 0x7f2c6019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c601094c0 0x7f2c60199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 --2- 172.21.15.1:0/3764125181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60103e30 0x7f2c6019fd00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 >> 172.21.15.1:0/3764125181 conn(0x7f2c6007a580 msgr2=0x7f2c60102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 shutdown_connections 2024-04-26T22:48:57.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:48:57.049+0000 7f2c6526e700 1 -- 172.21.15.1:0/3764125181 wait complete. 2024-04-26T22:48:57.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:56 smithi001 ceph-mon[29435]: pgmap v330: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:57.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:56 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1044773438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T22:48:57.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:56 smithi186 ceph-mon[32774]: pgmap v330: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:57.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:56 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1044773438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T22:48:57.252 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:56 smithi038 ceph-mon[32433]: pgmap v330: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:48:57.253 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:56 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1044773438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T22:48:57.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182465 got 107374182471 for osd.3 2024-04-26T22:48:57.726 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:58.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:57 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3764125181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T22:48:58.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383703 got 343597383709 for osd.11 2024-04-26T22:48:58.189 DEBUG:teuthology.parallel:result is None 2024-04-26T22:48:58.189 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T22:48:58.190 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T22:48:58.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:57 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3764125181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T22:48:58.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:57 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3764125181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T22:48:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:48:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:59 smithi186 ceph-mon[32774]: pgmap v331: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:48:59 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:59.828 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:59 smithi001 ceph-mon[29435]: pgmap v331: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:59.828 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:59.828 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:48:59 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:59 smithi038 ceph-mon[32433]: pgmap v331: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:48:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:48:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:48:59 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:49:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/1157034792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 msgr2=0x7fcb98110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/1157034792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98110290 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000b0a0 tx=0x7fcb9000b3b0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/1157034792 shutdown_connections 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/1157034792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb981094c0 0x7fcb981098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/1157034792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98110290 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/1157034792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb98103470 0x7fcb981038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/1157034792 >> 172.21.15.1:0/1157034792 conn(0x7fcb9807a580 msgr2=0x7fcb9807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/1157034792 shutdown_connections 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/1157034792 wait complete. 2024-04-26T22:49:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 Processor -- start 2024-04-26T22:49:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.220+0000 7fcb9ec3d700 1 -- start start 2024-04-26T22:49:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9ec3d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb98103470 0x7fcb98075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9ec3d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9ec3d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb981094c0 0x7fcb980761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9ec3d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb981135e0 con 0x7fcb98103e30 2024-04-26T22:49:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9ec3d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb98113460 con 0x7fcb98103470 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9ec3d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb98113760 con 0x7fcb981094c0 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52754/0 (socket says 172.21.15.1:52754) 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 -- 172.21.15.1:0/3292537588 learned_addr learned my addr 172.21.15.1:0/3292537588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9d1da700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb981094c0 0x7fcb980761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb9c9d9700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb98103470 0x7fcb98075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 -- 172.21.15.1:0/3292537588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb981094c0 msgr2=0x7fcb980761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb981094c0 0x7fcb980761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 -- 172.21.15.1:0/3292537588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb98103470 msgr2=0x7fcb98075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb98103470 0x7fcb98075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 -- 172.21.15.1:0/3292537588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb981a7f40 con 0x7fcb98103e30 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.221+0000 7fcb97fff700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98075c60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c9b0 tx=0x7fcb8400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.222+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb84007b00 con 0x7fcb98103e30 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.222+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb981a8230 con 0x7fcb98103e30 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.222+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb84004ce0 con 0x7fcb98103e30 2024-04-26T22:49:00.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.222+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb84005530 con 0x7fcb98103e30 2024-04-26T22:49:00.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.222+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb981a87c0 con 0x7fcb98103e30 2024-04-26T22:49:00.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.223+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb9806a110 con 0x7fcb98103e30 2024-04-26T22:49:00.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.224+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fcb84004750 con 0x7fcb98103e30 2024-04-26T22:49:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.224+0000 7fcb95ffb700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb880a3cd0 0x7fcb880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.225+0000 7fcb9c9d9700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb880a3cd0 0x7fcb880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.225+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fcb840d05b0 con 0x7fcb98103e30 2024-04-26T22:49:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.225+0000 7fcb9c9d9700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb880a3cd0 0x7fcb880a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00bbb0 tx=0x7fcb8c00b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:00.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.229+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcb84014030 con 0x7fcb98103e30 2024-04-26T22:49:00.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.434+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb98063860 con 0x7fcb880a3cd0 2024-04-26T22:49:00.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.441+0000 7fcb95ffb700 1 -- 172.21.15.1:0/3292537588 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258980 (secure 0 0 0) 0x7fcb98063860 con 0x7fcb880a3cd0 2024-04-26T22:49:00.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:00.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.444+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb880a3cd0 msgr2=0x7fcb880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:00.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.444+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb880a3cd0 0x7fcb880a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00bbb0 tx=0x7fcb8c00b590 comp rx=0 tx=0).stop 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.444+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 msgr2=0x7fcb98075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.444+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98075c60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c9b0 tx=0x7fcb8400ccc0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 shutdown_connections 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcb880a3cd0 0x7fcb880a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb981094c0 0x7fcb980761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb98103470 0x7fcb98075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 --2- 172.21.15.1:0/3292537588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb98103e30 0x7fcb98075c60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 >> 172.21.15.1:0/3292537588 conn(0x7fcb9807a580 msgr2=0x7fcb981020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 shutdown_connections 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:00.445+0000 7fcb9ec3d700 1 -- 172.21.15.1:0/3292537588 wait complete. 2024-04-26T22:49:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T22:49:00.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:00 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:49:00.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:00 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:49:00.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:00 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:49:00.805 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:00 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:49:00.806 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:00 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:49:00.806 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:00 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:49:00.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:00 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:49:00.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:00 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:49:00.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:00 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:49:01.175 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-04-26T22:48:59.769671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648204,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":860,"num_read_kb":538,"num_write":415,"num_write_kb":3061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360904,"kb_used_data":11676,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580920,"statfs":{"total":1151940427776,"available":1151570862080,"internally_reserved":0,"allocated":11956224,"data_stored":7411044,"data_compressed":115386,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922625+0000","last_change":"2024-04-26T22:47:41.777587+0000","last_active":"2024-04-26T22:48:30.922625+0000","last_peered":"2024-04-26T22:48:30.922625+0000","last_clean":"2024-04-26T22:48:30.922625+0000","last_became_active":"2024-04-26T22:47:41.777165+0000","last_became_peered":"2024-04-26T22:47:41.777165+0000","last_unstale":"2024-04-26T22:48:30.922625+0000","last_undegraded":"2024-04-26T22:48:30.922625+0000","last_fullsized":"2024-04-26T22:48:30.922625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:45.131599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919933+0000","last_change":"2024-04-26T22:47:41.776888+0000","last_active":"2024-04-26T22:48:30.919933+0000","last_peered":"2024-04-26T22:48:30.919933+0000","last_clean":"2024-04-26T22:48:30.919933+0000","last_became_active":"2024-04-26T22:47:41.776580+0000","last_became_peered":"2024-04-26T22:47:41.776580+0000","last_unstale":"2024-04-26T22:48:30.919933+0000","last_undegraded":"2024-04-26T22:48:30.919933+0000","last_fullsized":"2024-04-26T22:48:30.919933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:49.263390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919709+0000","last_change":"2024-04-26T22:47:41.784374+0000","last_active":"2024-04-26T22:48:30.919709+0000","last_peered":"2024-04-26T22:48:30.919709+0000","last_clean":"2024-04-26T22:48:30.919709+0000","last_became_active":"2024-04-26T22:47:41.783119+0000","last_became_peered":"2024-04-26T22:47:41.783119+0000","last_unstale":"2024-04-26T22:48:30.919709+0000","last_undegraded":"2024-04-26T22:48:30.919709+0000","last_fullsized":"2024-04-26T22:48:30.919709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:18.599515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924713+0000","last_change":"2024-04-26T22:47:41.782485+0000","last_active":"2024-04-26T22:48:30.924713+0000","last_peered":"2024-04-26T22:48:30.924713+0000","last_clean":"2024-04-26T22:48:30.924713+0000","last_became_active":"2024-04-26T22:47:41.782118+0000","last_became_peered":"2024-04-26T22:47:41.782118+0000","last_unstale":"2024-04-26T22:48:30.924713+0000","last_undegraded":"2024-04-26T22:48:30.924713+0000","last_fullsized":"2024-04-26T22:48:30.924713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:39.953297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919757+0000","last_change":"2024-04-26T22:47:41.776281+0000","last_active":"2024-04-26T22:48:30.919757+0000","last_peered":"2024-04-26T22:48:30.919757+0000","last_clean":"2024-04-26T22:48:30.919757+0000","last_became_active":"2024-04-26T22:47:41.775838+0000","last_became_peered":"2024-04-26T22:47:41.775838+0000","last_unstale":"2024-04-26T22:48:30.919757+0000","last_undegraded":"2024-04-26T22:48:30.919757+0000","last_fullsized":"2024-04-26T22:48:30.919757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:04.702698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993201+0000","last_change":"2024-04-26T22:47:41.785568+0000","last_active":"2024-04-26T22:48:30.993201+0000","last_peered":"2024-04-26T22:48:30.993201+0000","last_clean":"2024-04-26T22:48:30.993201+0000","last_became_active":"2024-04-26T22:47:41.785188+0000","last_became_peered":"2024-04-26T22:47:41.785188+0000","last_unstale":"2024-04-26T22:48:30.993201+0000","last_undegraded":"2024-04-26T22:48:30.993201+0000","last_fullsized":"2024-04-26T22:48:30.993201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:21.928641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927020+0000","last_change":"2024-04-26T22:47:41.785722+0000","last_active":"2024-04-26T22:48:30.927020+0000","last_peered":"2024-04-26T22:48:30.927020+0000","last_clean":"2024-04-26T22:48:30.927020+0000","last_became_active":"2024-04-26T22:47:41.784935+0000","last_became_peered":"2024-04-26T22:47:41.784935+0000","last_unstale":"2024-04-26T22:48:30.927020+0000","last_undegraded":"2024-04-26T22:48:30.927020+0000","last_fullsized":"2024-04-26T22:48:30.927020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:24.047598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.994932+0000","last_change":"2024-04-26T22:48:30.994932+0000","last_active":"2024-04-26T22:48:30.994932+0000","last_peered":"2024-04-26T22:48:30.994932+0000","last_clean":"2024-04-26T22:48:30.994932+0000","last_became_active":"2024-04-26T22:48:30.994471+0000","last_became_peered":"2024-04-26T22:48:30.994471+0000","last_unstale":"2024-04-26T22:48:30.994932+0000","last_undegraded":"2024-04-26T22:48:30.994932+0000","last_fullsized":"2024-04-26T22:48:30.994932+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:13:08.678628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924764+0000","last_change":"2024-04-26T22:47:41.775207+0000","last_active":"2024-04-26T22:48:30.924764+0000","last_peered":"2024-04-26T22:48:30.924764+0000","last_clean":"2024-04-26T22:48:30.924764+0000","last_became_active":"2024-04-26T22:47:41.774834+0000","last_became_peered":"2024-04-26T22:47:41.774834+0000","last_unstale":"2024-04-26T22:48:30.924764+0000","last_undegraded":"2024-04-26T22:48:30.924764+0000","last_fullsized":"2024-04-26T22:48:30.924764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:25.913385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993516+0000","last_change":"2024-04-26T22:47:41.785894+0000","last_active":"2024-04-26T22:48:30.993516+0000","last_peered":"2024-04-26T22:48:30.993516+0000","last_clean":"2024-04-26T22:48:30.993516+0000","last_became_active":"2024-04-26T22:47:41.785213+0000","last_became_peered":"2024-04-26T22:47:41.785213+0000","last_unstale":"2024-04-26T22:48:30.993516+0000","last_undegraded":"2024-04-26T22:48:30.993516+0000","last_fullsized":"2024-04-26T22:48:30.993516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:44.927776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927464+0000","last_change":"2024-04-26T22:47:41.781532+0000","last_active":"2024-04-26T22:48:30.927464+0000","last_peered":"2024-04-26T22:48:30.927464+0000","last_clean":"2024-04-26T22:48:30.927464+0000","last_became_active":"2024-04-26T22:47:41.781111+0000","last_became_peered":"2024-04-26T22:47:41.781111+0000","last_unstale":"2024-04-26T22:48:30.927464+0000","last_undegraded":"2024-04-26T22:48:30.927464+0000","last_fullsized":"2024-04-26T22:48:30.927464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:54.930233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924644+0000","last_change":"2024-04-26T22:47:41.782289+0000","last_active":"2024-04-26T22:48:30.924644+0000","last_peered":"2024-04-26T22:48:30.924644+0000","last_clean":"2024-04-26T22:48:30.924644+0000","last_became_active":"2024-04-26T22:47:41.781922+0000","last_became_peered":"2024-04-26T22:47:41.781922+0000","last_unstale":"2024-04-26T22:48:30.924644+0000","last_undegraded":"2024-04-26T22:48:30.924644+0000","last_fullsized":"2024-04-26T22:48:30.924644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:30.146485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993630+0000","last_change":"2024-04-26T22:47:41.770970+0000","last_active":"2024-04-26T22:48:30.993630+0000","last_peered":"2024-04-26T22:48:30.993630+0000","last_clean":"2024-04-26T22:48:30.993630+0000","last_became_active":"2024-04-26T22:47:41.770502+0000","last_became_peered":"2024-04-26T22:47:41.770502+0000","last_unstale":"2024-04-26T22:48:30.993630+0000","last_undegraded":"2024-04-26T22:48:30.993630+0000","last_fullsized":"2024-04-26T22:48:30.993630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.988613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919582+0000","last_change":"2024-04-26T22:47:41.779941+0000","last_active":"2024-04-26T22:48:30.919582+0000","last_peered":"2024-04-26T22:48:30.919582+0000","last_clean":"2024-04-26T22:48:30.919582+0000","last_became_active":"2024-04-26T22:47:41.779592+0000","last_became_peered":"2024-04-26T22:47:41.779592+0000","last_unstale":"2024-04-26T22:48:30.919582+0000","last_undegraded":"2024-04-26T22:48:30.919582+0000","last_fullsized":"2024-04-26T22:48:30.919582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:58.628553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.995248+0000","last_change":"2024-04-26T22:48:30.995248+0000","last_active":"2024-04-26T22:48:30.995248+0000","last_peered":"2024-04-26T22:48:30.995248+0000","last_clean":"2024-04-26T22:48:30.995248+0000","last_became_active":"2024-04-26T22:48:30.994823+0000","last_became_peered":"2024-04-26T22:48:30.994823+0000","last_unstale":"2024-04-26T22:48:30.995248+0000","last_undegraded":"2024-04-26T22:48:30.995248+0000","last_fullsized":"2024-04-26T22:48:30.995248+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:51:12.855303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923528+0000","last_change":"2024-04-26T22:47:41.771277+0000","last_active":"2024-04-26T22:48:30.923528+0000","last_peered":"2024-04-26T22:48:30.923528+0000","last_clean":"2024-04-26T22:48:30.923528+0000","last_became_active":"2024-04-26T22:47:41.770933+0000","last_became_peered":"2024-04-26T22:47:41.770933+0000","last_unstale":"2024-04-26T22:48:30.923528+0000","last_undegraded":"2024-04-26T22:48:30.923528+0000","last_fullsized":"2024-04-26T22:48:30.923528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:15.045678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922368+0000","last_change":"2024-04-26T22:47:41.778656+0000","last_active":"2024-04-26T22:48:30.922368+0000","last_peered":"2024-04-26T22:48:30.922368+0000","last_clean":"2024-04-26T22:48:30.922368+0000","last_became_active":"2024-04-26T22:47:41.778282+0000","last_became_peered":"2024-04-26T22:47:41.778282+0000","last_unstale":"2024-04-26T22:48:30.922368+0000","last_undegraded":"2024-04-26T22:48:30.922368+0000","last_fullsized":"2024-04-26T22:48:30.922368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:12.987690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922065+0000","last_change":"2024-04-26T22:47:41.776965+0000","last_active":"2024-04-26T22:48:30.922065+0000","last_peered":"2024-04-26T22:48:30.922065+0000","last_clean":"2024-04-26T22:48:30.922065+0000","last_became_active":"2024-04-26T22:47:41.776625+0000","last_became_peered":"2024-04-26T22:47:41.776625+0000","last_unstale":"2024-04-26T22:48:30.922065+0000","last_undegraded":"2024-04-26T22:48:30.922065+0000","last_fullsized":"2024-04-26T22:48:30.922065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:38.601785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927130+0000","last_change":"2024-04-26T22:47:41.796742+0000","last_active":"2024-04-26T22:48:30.927130+0000","last_peered":"2024-04-26T22:48:30.927130+0000","last_clean":"2024-04-26T22:48:30.927130+0000","last_became_active":"2024-04-26T22:47:41.796362+0000","last_became_peered":"2024-04-26T22:47:41.796362+0000","last_unstale":"2024-04-26T22:48:30.927130+0000","last_undegraded":"2024-04-26T22:48:30.927130+0000","last_fullsized":"2024-04-26T22:48:30.927130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:01:55.092075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:53.905197+0000","last_change":"2024-04-26T22:47:40.771619+0000","last_active":"2024-04-26T22:48:53.905197+0000","last_peered":"2024-04-26T22:48:53.905197+0000","last_clean":"2024-04-26T22:48:53.905197+0000","last_became_active":"2024-04-26T22:47:40.771262+0000","last_became_peered":"2024-04-26T22:47:40.771262+0000","last_unstale":"2024-04-26T22:48:53.905197+0000","last_undegraded":"2024-04-26T22:48:53.905197+0000","last_fullsized":"2024-04-26T22:48:53.905197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:19.528487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993392+0000","last_change":"2024-04-26T22:47:41.784162+0000","last_active":"2024-04-26T22:48:30.993392+0000","last_peered":"2024-04-26T22:48:30.993392+0000","last_clean":"2024-04-26T22:48:30.993392+0000","last_became_active":"2024-04-26T22:47:41.783821+0000","last_became_peered":"2024-04-26T22:47:41.783821+0000","last_unstale":"2024-04-26T22:48:30.993392+0000","last_undegraded":"2024-04-26T22:48:30.993392+0000","last_fullsized":"2024-04-26T22:48:30.993392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:17:04.237927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993480+0000","last_change":"2024-04-26T22:47:40.779858+0000","last_active":"2024-04-26T22:48:30.993480+0000","last_peered":"2024-04-26T22:48:30.993480+0000","last_clean":"2024-04-26T22:48:30.993480+0000","last_became_active":"2024-04-26T22:47:40.779550+0000","last_became_peered":"2024-04-26T22:47:40.779550+0000","last_unstale":"2024-04-26T22:48:30.993480+0000","last_undegraded":"2024-04-26T22:48:30.993480+0000","last_fullsized":"2024-04-26T22:48:30.993480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:40:54.056426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919942+0000","last_change":"2024-04-26T22:47:41.769865+0000","last_active":"2024-04-26T22:48:30.919942+0000","last_peered":"2024-04-26T22:48:30.919942+0000","last_clean":"2024-04-26T22:48:30.919942+0000","last_became_active":"2024-04-26T22:47:41.769545+0000","last_became_peered":"2024-04-26T22:47:41.769545+0000","last_unstale":"2024-04-26T22:48:30.919942+0000","last_undegraded":"2024-04-26T22:48:30.919942+0000","last_fullsized":"2024-04-26T22:48:30.919942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:39.715241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920239+0000","last_change":"2024-04-26T22:47:40.771234+0000","last_active":"2024-04-26T22:48:30.920239+0000","last_peered":"2024-04-26T22:48:30.920239+0000","last_clean":"2024-04-26T22:48:30.920239+0000","last_became_active":"2024-04-26T22:47:40.770876+0000","last_became_peered":"2024-04-26T22:47:40.770876+0000","last_unstale":"2024-04-26T22:48:30.920239+0000","last_undegraded":"2024-04-26T22:48:30.920239+0000","last_fullsized":"2024-04-26T22:48:30.920239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:50:41.272029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923627+0000","last_change":"2024-04-26T22:47:41.770470+0000","last_active":"2024-04-26T22:48:30.923627+0000","last_peered":"2024-04-26T22:48:30.923627+0000","last_clean":"2024-04-26T22:48:30.923627+0000","last_became_active":"2024-04-26T22:47:41.767318+0000","last_became_peered":"2024-04-26T22:47:41.767318+0000","last_unstale":"2024-04-26T22:48:30.923627+0000","last_undegraded":"2024-04-26T22:48:30.923627+0000","last_fullsized":"2024-04-26T22:48:30.923627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:45.324118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920119+0000","last_change":"2024-04-26T22:47:40.775811+0000","last_active":"2024-04-26T22:48:30.920119+0000","last_peered":"2024-04-26T22:48:30.920119+0000","last_clean":"2024-04-26T22:48:30.920119+0000","last_became_active":"2024-04-26T22:47:40.775349+0000","last_became_peered":"2024-04-26T22:47:40.775349+0000","last_unstale":"2024-04-26T22:48:30.920119+0000","last_undegraded":"2024-04-26T22:48:30.920119+0000","last_fullsized":"2024-04-26T22:48:30.920119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.926983+0000","last_change":"2024-04-26T22:48:30.926983+0000","last_active":"2024-04-26T22:48:30.926983+0000","last_peered":"2024-04-26T22:48:30.926983+0000","last_clean":"2024-04-26T22:48:30.926983+0000","last_became_active":"2024-04-26T22:48:30.926588+0000","last_became_peered":"2024-04-26T22:48:30.926588+0000","last_unstale":"2024-04-26T22:48:30.926983+0000","last_undegraded":"2024-04-26T22:48:30.926983+0000","last_fullsized":"2024-04-26T22:48:30.926983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:51:09.309076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:53.898798+0000","last_change":"2024-04-26T22:47:40.774855+0000","last_active":"2024-04-26T22:48:53.898798+0000","last_peered":"2024-04-26T22:48:53.898798+0000","last_clean":"2024-04-26T22:48:53.898798+0000","last_became_active":"2024-04-26T22:47:40.774517+0000","last_became_peered":"2024-04-26T22:47:40.774517+0000","last_unstale":"2024-04-26T22:48:53.898798+0000","last_undegraded":"2024-04-26T22:48:53.898798+0000","last_fullsized":"2024-04-26T22:48:53.898798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.811072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919963+0000","last_change":"2024-04-26T22:47:41.781432+0000","last_active":"2024-04-26T22:48:30.919963+0000","last_peered":"2024-04-26T22:48:30.919963+0000","last_clean":"2024-04-26T22:48:30.919963+0000","last_became_active":"2024-04-26T22:47:41.781035+0000","last_became_peered":"2024-04-26T22:47:41.781035+0000","last_unstale":"2024-04-26T22:48:30.919963+0000","last_undegraded":"2024-04-26T22:48:30.919963+0000","last_fullsized":"2024-04-26T22:48:30.919963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:31.730087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993235+0000","last_change":"2024-04-26T22:47:40.770419+0000","last_active":"2024-04-26T22:48:30.993235+0000","last_peered":"2024-04-26T22:48:30.993235+0000","last_clean":"2024-04-26T22:48:30.993235+0000","last_became_active":"2024-04-26T22:47:40.769923+0000","last_became_peered":"2024-04-26T22:47:40.769923+0000","last_unstale":"2024-04-26T22:48:30.993235+0000","last_undegraded":"2024-04-26T22:48:30.993235+0000","last_fullsized":"2024-04-26T22:48:30.993235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:15.618904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924075+0000","last_change":"2024-04-26T22:47:41.778334+0000","last_active":"2024-04-26T22:48:30.924075+0000","last_peered":"2024-04-26T22:48:30.924075+0000","last_clean":"2024-04-26T22:48:30.924075+0000","last_became_active":"2024-04-26T22:47:41.777822+0000","last_became_peered":"2024-04-26T22:47:41.777822+0000","last_unstale":"2024-04-26T22:48:30.924075+0000","last_undegraded":"2024-04-26T22:48:30.924075+0000","last_fullsized":"2024-04-26T22:48:30.924075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:57.828165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993903+0000","last_change":"2024-04-26T22:47:40.779524+0000","last_active":"2024-04-26T22:48:30.993903+0000","last_peered":"2024-04-26T22:48:30.993903+0000","last_clean":"2024-04-26T22:48:30.993903+0000","last_became_active":"2024-04-26T22:47:40.778998+0000","last_became_peered":"2024-04-26T22:47:40.778998+0000","last_unstale":"2024-04-26T22:48:30.993903+0000","last_undegraded":"2024-04-26T22:48:30.993903+0000","last_fullsized":"2024-04-26T22:48:30.993903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:59.979648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992671+0000","last_change":"2024-04-26T22:47:41.768883+0000","last_active":"2024-04-26T22:48:30.992671+0000","last_peered":"2024-04-26T22:48:30.992671+0000","last_clean":"2024-04-26T22:48:30.992671+0000","last_became_active":"2024-04-26T22:47:41.768577+0000","last_became_peered":"2024-04-26T22:47:41.768577+0000","last_unstale":"2024-04-26T22:48:30.992671+0000","last_undegraded":"2024-04-26T22:48:30.992671+0000","last_fullsized":"2024-04-26T22:48:30.992671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:29.267873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923845+0000","last_change":"2024-04-26T22:47:40.774688+0000","last_active":"2024-04-26T22:48:30.923845+0000","last_peered":"2024-04-26T22:48:30.923845+0000","last_clean":"2024-04-26T22:48:30.923845+0000","last_became_active":"2024-04-26T22:47:40.774312+0000","last_became_peered":"2024-04-26T22:47:40.774312+0000","last_unstale":"2024-04-26T22:48:30.923845+0000","last_undegraded":"2024-04-26T22:48:30.923845+0000","last_fullsized":"2024-04-26T22:48:30.923845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:03:34.797965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921763+0000","last_change":"2024-04-26T22:47:41.776948+0000","last_active":"2024-04-26T22:48:30.921763+0000","last_peered":"2024-04-26T22:48:30.921763+0000","last_clean":"2024-04-26T22:48:30.921763+0000","last_became_active":"2024-04-26T22:47:41.776583+0000","last_became_peered":"2024-04-26T22:47:41.776583+0000","last_unstale":"2024-04-26T22:48:30.921763+0000","last_undegraded":"2024-04-26T22:48:30.921763+0000","last_fullsized":"2024-04-26T22:48:30.921763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:42.687249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":163,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:55.900991+0000","last_change":"2024-04-26T22:47:40.772060+0000","last_active":"2024-04-26T22:48:55.900991+0000","last_peered":"2024-04-26T22:48:55.900991+0000","last_clean":"2024-04-26T22:48:55.900991+0000","last_became_active":"2024-04-26T22:47:40.771690+0000","last_became_peered":"2024-04-26T22:47:40.771690+0000","last_unstale":"2024-04-26T22:48:55.900991+0000","last_undegraded":"2024-04-26T22:48:55.900991+0000","last_fullsized":"2024-04-26T22:48:55.900991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:53.619399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924433+0000","last_change":"2024-04-26T22:47:41.776902+0000","last_active":"2024-04-26T22:48:30.924433+0000","last_peered":"2024-04-26T22:48:30.924433+0000","last_clean":"2024-04-26T22:48:30.924433+0000","last_became_active":"2024-04-26T22:47:41.776475+0000","last_became_peered":"2024-04-26T22:47:41.776475+0000","last_unstale":"2024-04-26T22:48:30.924433+0000","last_undegraded":"2024-04-26T22:48:30.924433+0000","last_fullsized":"2024-04-26T22:48:30.924433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:57:17.674715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927603+0000","last_change":"2024-04-26T22:47:40.775828+0000","last_active":"2024-04-26T22:48:30.927603+0000","last_peered":"2024-04-26T22:48:30.927603+0000","last_clean":"2024-04-26T22:48:30.927603+0000","last_became_active":"2024-04-26T22:47:40.775439+0000","last_became_peered":"2024-04-26T22:47:40.775439+0000","last_unstale":"2024-04-26T22:48:30.927603+0000","last_undegraded":"2024-04-26T22:48:30.927603+0000","last_fullsized":"2024-04-26T22:48:30.927603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:28:21.061843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919938+0000","last_change":"2024-04-26T22:47:41.782731+0000","last_active":"2024-04-26T22:48:30.919938+0000","last_peered":"2024-04-26T22:48:30.919938+0000","last_clean":"2024-04-26T22:48:30.919938+0000","last_became_active":"2024-04-26T22:47:41.781390+0000","last_became_peered":"2024-04-26T22:47:41.781390+0000","last_unstale":"2024-04-26T22:48:30.919938+0000","last_undegraded":"2024-04-26T22:48:30.919938+0000","last_fullsized":"2024-04-26T22:48:30.919938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:15.349252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924169+0000","last_change":"2024-04-26T22:47:40.770488+0000","last_active":"2024-04-26T22:48:30.924169+0000","last_peered":"2024-04-26T22:48:30.924169+0000","last_clean":"2024-04-26T22:48:30.924169+0000","last_became_active":"2024-04-26T22:47:40.770049+0000","last_became_peered":"2024-04-26T22:47:40.770049+0000","last_unstale":"2024-04-26T22:48:30.924169+0000","last_undegraded":"2024-04-26T22:48:30.924169+0000","last_fullsized":"2024-04-26T22:48:30.924169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:03.391917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923853+0000","last_change":"2024-04-26T22:47:41.778433+0000","last_active":"2024-04-26T22:48:30.923853+0000","last_peered":"2024-04-26T22:48:30.923853+0000","last_clean":"2024-04-26T22:48:30.923853+0000","last_became_active":"2024-04-26T22:47:41.777603+0000","last_became_peered":"2024-04-26T22:47:41.777603+0000","last_unstale":"2024-04-26T22:48:30.923853+0000","last_undegraded":"2024-04-26T22:48:30.923853+0000","last_fullsized":"2024-04-26T22:48:30.923853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:02.206291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'27","reported_seq":192,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:55.900674+0000","last_change":"2024-04-26T22:47:40.774417+0000","last_active":"2024-04-26T22:48:55.900674+0000","last_peered":"2024-04-26T22:48:55.900674+0000","last_clean":"2024-04-26T22:48:55.900674+0000","last_became_active":"2024-04-26T22:47:40.773932+0000","last_became_peered":"2024-04-26T22:47:40.773932+0000","last_unstale":"2024-04-26T22:48:55.900674+0000","last_undegraded":"2024-04-26T22:48:55.900674+0000","last_fullsized":"2024-04-26T22:48:55.900674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:15.694401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50398,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":146,"num_read_kb":122,"num_write":29,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920014+0000","last_change":"2024-04-26T22:47:41.787466+0000","last_active":"2024-04-26T22:48:30.920014+0000","last_peered":"2024-04-26T22:48:30.920014+0000","last_clean":"2024-04-26T22:48:30.920014+0000","last_became_active":"2024-04-26T22:47:41.787105+0000","last_became_peered":"2024-04-26T22:47:41.787105+0000","last_unstale":"2024-04-26T22:48:30.920014+0000","last_undegraded":"2024-04-26T22:48:30.920014+0000","last_fullsized":"2024-04-26T22:48:30.920014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:48:34.656495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927272+0000","last_change":"2024-04-26T22:47:40.767810+0000","last_active":"2024-04-26T22:48:30.927272+0000","last_peered":"2024-04-26T22:48:30.927272+0000","last_clean":"2024-04-26T22:48:30.927272+0000","last_became_active":"2024-04-26T22:47:40.767508+0000","last_became_peered":"2024-04-26T22:47:40.767508+0000","last_unstale":"2024-04-26T22:48:30.927272+0000","last_undegraded":"2024-04-26T22:48:30.927272+0000","last_fullsized":"2024-04-26T22:48:30.927272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:32.023939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993557+0000","last_change":"2024-04-26T22:47:41.774134+0000","last_active":"2024-04-26T22:48:30.993557+0000","last_peered":"2024-04-26T22:48:30.993557+0000","last_clean":"2024-04-26T22:48:30.993557+0000","last_became_active":"2024-04-26T22:47:41.773613+0000","last_became_peered":"2024-04-26T22:47:41.773613+0000","last_unstale":"2024-04-26T22:48:30.993557+0000","last_undegraded":"2024-04-26T22:48:30.993557+0000","last_fullsized":"2024-04-26T22:48:30.993557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:41:51.317633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923756+0000","last_change":"2024-04-26T22:47:40.776836+0000","last_active":"2024-04-26T22:48:30.923756+0000","last_peered":"2024-04-26T22:48:30.923756+0000","last_clean":"2024-04-26T22:48:30.923756+0000","last_became_active":"2024-04-26T22:47:40.776493+0000","last_became_peered":"2024-04-26T22:47:40.776493+0000","last_unstale":"2024-04-26T22:48:30.923756+0000","last_undegraded":"2024-04-26T22:48:30.923756+0000","last_fullsized":"2024-04-26T22:48:30.923756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:55.695456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.995002+0000","last_change":"2024-04-26T22:48:30.995002+0000","last_active":"2024-04-26T22:48:30.995002+0000","last_peered":"2024-04-26T22:48:30.995002+0000","last_clean":"2024-04-26T22:48:30.995002+0000","last_became_active":"2024-04-26T22:48:30.994655+0000","last_became_peered":"2024-04-26T22:48:30.994655+0000","last_unstale":"2024-04-26T22:48:30.995002+0000","last_undegraded":"2024-04-26T22:48:30.995002+0000","last_fullsized":"2024-04-26T22:48:30.995002+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:00.209207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:57.904130+0000","last_change":"2024-04-26T22:47:40.768984+0000","last_active":"2024-04-26T22:48:57.904130+0000","last_peered":"2024-04-26T22:48:57.904130+0000","last_clean":"2024-04-26T22:48:57.904130+0000","last_became_active":"2024-04-26T22:47:40.768596+0000","last_became_peered":"2024-04-26T22:47:40.768596+0000","last_unstale":"2024-04-26T22:48:57.904130+0000","last_undegraded":"2024-04-26T22:48:57.904130+0000","last_fullsized":"2024-04-26T22:48:57.904130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:44.099790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927308+0000","last_change":"2024-04-26T22:47:41.778100+0000","last_active":"2024-04-26T22:48:30.927308+0000","last_peered":"2024-04-26T22:48:30.927308+0000","last_clean":"2024-04-26T22:48:30.927308+0000","last_became_active":"2024-04-26T22:47:41.777775+0000","last_became_peered":"2024-04-26T22:47:41.777775+0000","last_unstale":"2024-04-26T22:48:30.927308+0000","last_undegraded":"2024-04-26T22:48:30.927308+0000","last_fullsized":"2024-04-26T22:48:30.927308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:52.698089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919693+0000","last_change":"2024-04-26T22:47:40.761904+0000","last_active":"2024-04-26T22:48:30.919693+0000","last_peered":"2024-04-26T22:48:30.919693+0000","last_clean":"2024-04-26T22:48:30.919693+0000","last_became_active":"2024-04-26T22:47:40.761591+0000","last_became_peered":"2024-04-26T22:47:40.761591+0000","last_unstale":"2024-04-26T22:48:30.919693+0000","last_undegraded":"2024-04-26T22:48:30.919693+0000","last_fullsized":"2024-04-26T22:48:30.919693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:55.774669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923569+0000","last_change":"2024-04-26T22:47:41.780703+0000","last_active":"2024-04-26T22:48:30.923569+0000","last_peered":"2024-04-26T22:48:30.923569+0000","last_clean":"2024-04-26T22:48:30.923569+0000","last_became_active":"2024-04-26T22:47:41.775023+0000","last_became_peered":"2024-04-26T22:47:41.775023+0000","last_unstale":"2024-04-26T22:48:30.923569+0000","last_undegraded":"2024-04-26T22:48:30.923569+0000","last_fullsized":"2024-04-26T22:48:30.923569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:09.515514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.933400+0000","last_change":"2024-04-26T22:48:30.932931+0000","last_active":"2024-04-26T22:48:30.933400+0000","last_peered":"2024-04-26T22:48:30.933400+0000","last_clean":"2024-04-26T22:48:30.933400+0000","last_became_active":"2024-04-26T22:48:30.926196+0000","last_became_peered":"2024-04-26T22:48:30.926196+0000","last_unstale":"2024-04-26T22:48:30.933400+0000","last_undegraded":"2024-04-26T22:48:30.933400+0000","last_fullsized":"2024-04-26T22:48:30.933400+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:43:44.026285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922011+0000","last_change":"2024-04-26T22:47:41.781793+0000","last_active":"2024-04-26T22:48:30.922011+0000","last_peered":"2024-04-26T22:48:30.922011+0000","last_clean":"2024-04-26T22:48:30.922011+0000","last_became_active":"2024-04-26T22:47:41.781470+0000","last_became_peered":"2024-04-26T22:47:41.781470+0000","last_unstale":"2024-04-26T22:48:30.922011+0000","last_undegraded":"2024-04-26T22:48:30.922011+0000","last_fullsized":"2024-04-26T22:48:30.922011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:20.600909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921912+0000","last_change":"2024-04-26T22:47:40.767364+0000","last_active":"2024-04-26T22:48:30.921912+0000","last_peered":"2024-04-26T22:48:30.921912+0000","last_clean":"2024-04-26T22:48:30.921912+0000","last_became_active":"2024-04-26T22:47:40.766907+0000","last_became_peered":"2024-04-26T22:47:40.766907+0000","last_unstale":"2024-04-26T22:48:30.921912+0000","last_undegraded":"2024-04-26T22:48:30.921912+0000","last_fullsized":"2024-04-26T22:48:30.921912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:56.408996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993511+0000","last_change":"2024-04-26T22:47:41.784095+0000","last_active":"2024-04-26T22:48:30.993511+0000","last_peered":"2024-04-26T22:48:30.993511+0000","last_clean":"2024-04-26T22:48:30.993511+0000","last_became_active":"2024-04-26T22:47:41.783749+0000","last_became_peered":"2024-04-26T22:47:41.783749+0000","last_unstale":"2024-04-26T22:48:30.993511+0000","last_undegraded":"2024-04-26T22:48:30.993511+0000","last_fullsized":"2024-04-26T22:48:30.993511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:11:26.530223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:55.907717+0000","last_change":"2024-04-26T22:47:40.778988+0000","last_active":"2024-04-26T22:48:55.907717+0000","last_peered":"2024-04-26T22:48:55.907717+0000","last_clean":"2024-04-26T22:48:55.907717+0000","last_became_active":"2024-04-26T22:47:40.778686+0000","last_became_peered":"2024-04-26T22:47:40.778686+0000","last_unstale":"2024-04-26T22:48:55.907717+0000","last_undegraded":"2024-04-26T22:48:55.907717+0000","last_fullsized":"2024-04-26T22:48:55.907717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.525866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993231+0000","last_change":"2024-04-26T22:47:41.787168+0000","last_active":"2024-04-26T22:48:30.993231+0000","last_peered":"2024-04-26T22:48:30.993231+0000","last_clean":"2024-04-26T22:48:30.993231+0000","last_became_active":"2024-04-26T22:47:41.786811+0000","last_became_peered":"2024-04-26T22:47:41.786811+0000","last_unstale":"2024-04-26T22:48:30.993231+0000","last_undegraded":"2024-04-26T22:48:30.993231+0000","last_fullsized":"2024-04-26T22:48:30.993231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:34.578058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.929267+0000","last_change":"2024-04-26T22:48:30.929267+0000","last_active":"2024-04-26T22:48:30.929267+0000","last_peered":"2024-04-26T22:48:30.929267+0000","last_clean":"2024-04-26T22:48:30.929267+0000","last_became_active":"2024-04-26T22:48:30.928927+0000","last_became_peered":"2024-04-26T22:48:30.928927+0000","last_unstale":"2024-04-26T22:48:30.929267+0000","last_undegraded":"2024-04-26T22:48:30.929267+0000","last_fullsized":"2024-04-26T22:48:30.929267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:44.270701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:54.899394+0000","last_change":"2024-04-26T22:47:40.781706+0000","last_active":"2024-04-26T22:48:54.899394+0000","last_peered":"2024-04-26T22:48:54.899394+0000","last_clean":"2024-04-26T22:48:54.899394+0000","last_became_active":"2024-04-26T22:47:40.781014+0000","last_became_peered":"2024-04-26T22:47:40.781014+0000","last_unstale":"2024-04-26T22:48:54.899394+0000","last_undegraded":"2024-04-26T22:48:54.899394+0000","last_fullsized":"2024-04-26T22:48:54.899394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:17:16.523767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993026+0000","last_change":"2024-04-26T22:47:41.772852+0000","last_active":"2024-04-26T22:48:30.993026+0000","last_peered":"2024-04-26T22:48:30.993026+0000","last_clean":"2024-04-26T22:48:30.993026+0000","last_became_active":"2024-04-26T22:47:41.772502+0000","last_became_peered":"2024-04-26T22:47:41.772502+0000","last_unstale":"2024-04-26T22:48:30.993026+0000","last_undegraded":"2024-04-26T22:48:30.993026+0000","last_fullsized":"2024-04-26T22:48:30.993026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:36.870593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924870+0000","last_change":"2024-04-26T22:47:40.768151+0000","last_active":"2024-04-26T22:48:30.924870+0000","last_peered":"2024-04-26T22:48:30.924870+0000","last_clean":"2024-04-26T22:48:30.924870+0000","last_became_active":"2024-04-26T22:47:40.767756+0000","last_became_peered":"2024-04-26T22:47:40.767756+0000","last_unstale":"2024-04-26T22:48:30.924870+0000","last_undegraded":"2024-04-26T22:48:30.924870+0000","last_fullsized":"2024-04-26T22:48:30.924870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:38:09.059145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927569+0000","last_change":"2024-04-26T22:46:24.327741+0000","last_active":"2024-04-26T22:48:30.927569+0000","last_peered":"2024-04-26T22:48:30.927569+0000","last_clean":"2024-04-26T22:48:30.927569+0000","last_became_active":"2024-04-26T22:46:24.327280+0000","last_became_peered":"2024-04-26T22:46:24.327280+0000","last_unstale":"2024-04-26T22:48:30.927569+0000","last_undegraded":"2024-04-26T22:48:30.927569+0000","last_fullsized":"2024-04-26T22:48:30.927569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_clean_scrub_stamp":"2024-04-26T22:42:55.984240+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:47.094131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993665+0000","last_change":"2024-04-26T22:47:41.779195+0000","last_active":"2024-04-26T22:48:30.993665+0000","last_peered":"2024-04-26T22:48:30.993665+0000","last_clean":"2024-04-26T22:48:30.993665+0000","last_became_active":"2024-04-26T22:47:41.778682+0000","last_became_peered":"2024-04-26T22:47:41.778682+0000","last_unstale":"2024-04-26T22:48:30.993665+0000","last_undegraded":"2024-04-26T22:48:30.993665+0000","last_fullsized":"2024-04-26T22:48:30.993665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:37.983340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927918+0000","last_change":"2024-04-26T22:47:40.774817+0000","last_active":"2024-04-26T22:48:30.927918+0000","last_peered":"2024-04-26T22:48:30.927918+0000","last_clean":"2024-04-26T22:48:30.927918+0000","last_became_active":"2024-04-26T22:47:40.772660+0000","last_became_peered":"2024-04-26T22:47:40.772660+0000","last_unstale":"2024-04-26T22:48:30.927918+0000","last_undegraded":"2024-04-26T22:48:30.927918+0000","last_fullsized":"2024-04-26T22:48:30.927918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:30.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920130+0000","last_change":"2024-04-26T22:47:40.773595+0000","last_active":"2024-04-26T22:48:30.920130+0000","last_peered":"2024-04-26T22:48:30.920130+0000","last_clean":"2024-04-26T22:48:30.920130+0000","last_became_active":"2024-04-26T22:47:40.773199+0000","last_became_peered":"2024-04-26T22:47:40.773199+0000","last_unstale":"2024-04-26T22:48:30.920130+0000","last_undegraded":"2024-04-26T22:48:30.920130+0000","last_fullsized":"2024-04-26T22:48:30.920130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:19:31.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993950+0000","last_change":"2024-04-26T22:47:41.780428+0000","last_active":"2024-04-26T22:48:30.993950+0000","last_peered":"2024-04-26T22:48:30.993950+0000","last_clean":"2024-04-26T22:48:30.993950+0000","last_became_active":"2024-04-26T22:47:41.779468+0000","last_became_peered":"2024-04-26T22:47:41.779468+0000","last_unstale":"2024-04-26T22:48:30.993950+0000","last_undegraded":"2024-04-26T22:48:30.993950+0000","last_fullsized":"2024-04-26T22:48:30.993950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:05.067347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922003+0000","last_change":"2024-04-26T22:47:40.770624+0000","last_active":"2024-04-26T22:48:30.922003+0000","last_peered":"2024-04-26T22:48:30.922003+0000","last_clean":"2024-04-26T22:48:30.922003+0000","last_became_active":"2024-04-26T22:47:40.770250+0000","last_became_peered":"2024-04-26T22:47:40.770250+0000","last_unstale":"2024-04-26T22:48:30.922003+0000","last_undegraded":"2024-04-26T22:48:30.922003+0000","last_fullsized":"2024-04-26T22:48:30.922003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:16:53.527984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993345+0000","last_change":"2024-04-26T22:47:41.773667+0000","last_active":"2024-04-26T22:48:30.993345+0000","last_peered":"2024-04-26T22:48:30.993345+0000","last_clean":"2024-04-26T22:48:30.993345+0000","last_became_active":"2024-04-26T22:47:41.772876+0000","last_became_peered":"2024-04-26T22:47:41.772876+0000","last_unstale":"2024-04-26T22:48:30.993345+0000","last_undegraded":"2024-04-26T22:48:30.993345+0000","last_fullsized":"2024-04-26T22:48:30.993345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.363992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924937+0000","last_change":"2024-04-26T22:47:40.774090+0000","last_active":"2024-04-26T22:48:30.924937+0000","last_peered":"2024-04-26T22:48:30.924937+0000","last_clean":"2024-04-26T22:48:30.924937+0000","last_became_active":"2024-04-26T22:47:40.773543+0000","last_became_peered":"2024-04-26T22:47:40.773543+0000","last_unstale":"2024-04-26T22:48:30.924937+0000","last_undegraded":"2024-04-26T22:48:30.924937+0000","last_fullsized":"2024-04-26T22:48:30.924937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:17:06.907699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919990+0000","last_change":"2024-04-26T22:47:41.781231+0000","last_active":"2024-04-26T22:48:30.919990+0000","last_peered":"2024-04-26T22:48:30.919990+0000","last_clean":"2024-04-26T22:48:30.919990+0000","last_became_active":"2024-04-26T22:47:41.780837+0000","last_became_peered":"2024-04-26T22:47:41.780837+0000","last_unstale":"2024-04-26T22:48:30.919990+0000","last_undegraded":"2024-04-26T22:48:30.919990+0000","last_fullsized":"2024-04-26T22:48:30.919990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:24.491628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.930700+0000","last_change":"2024-04-26T22:48:30.930700+0000","last_active":"2024-04-26T22:48:30.930700+0000","last_peered":"2024-04-26T22:48:30.930700+0000","last_clean":"2024-04-26T22:48:30.930700+0000","last_became_active":"2024-04-26T22:48:30.925999+0000","last_became_peered":"2024-04-26T22:48:30.925999+0000","last_unstale":"2024-04-26T22:48:30.930700+0000","last_undegraded":"2024-04-26T22:48:30.930700+0000","last_fullsized":"2024-04-26T22:48:30.930700+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:46.912066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919715+0000","last_change":"2024-04-26T22:47:41.776683+0000","last_active":"2024-04-26T22:48:30.919715+0000","last_peered":"2024-04-26T22:48:30.919715+0000","last_clean":"2024-04-26T22:48:30.919715+0000","last_became_active":"2024-04-26T22:47:41.776309+0000","last_became_peered":"2024-04-26T22:47:41.776309+0000","last_unstale":"2024-04-26T22:48:30.919715+0000","last_undegraded":"2024-04-26T22:48:30.919715+0000","last_fullsized":"2024-04-26T22:48:30.919715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:45.604270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992790+0000","last_change":"2024-04-26T22:47:40.774392+0000","last_active":"2024-04-26T22:48:30.992790+0000","last_peered":"2024-04-26T22:48:30.992790+0000","last_clean":"2024-04-26T22:48:30.992790+0000","last_became_active":"2024-04-26T22:47:40.773682+0000","last_became_peered":"2024-04-26T22:47:40.773682+0000","last_unstale":"2024-04-26T22:48:30.992790+0000","last_undegraded":"2024-04-26T22:48:30.992790+0000","last_fullsized":"2024-04-26T22:48:30.992790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:06:56.769791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921620+0000","last_change":"2024-04-26T22:47:41.784191+0000","last_active":"2024-04-26T22:48:30.921620+0000","last_peered":"2024-04-26T22:48:30.921620+0000","last_clean":"2024-04-26T22:48:30.921620+0000","last_became_active":"2024-04-26T22:47:41.783884+0000","last_became_peered":"2024-04-26T22:47:41.783884+0000","last_unstale":"2024-04-26T22:48:30.921620+0000","last_undegraded":"2024-04-26T22:48:30.921620+0000","last_fullsized":"2024-04-26T22:48:30.921620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:34.035506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927373+0000","last_change":"2024-04-26T22:47:40.767796+0000","last_active":"2024-04-26T22:48:30.927373+0000","last_peered":"2024-04-26T22:48:30.927373+0000","last_clean":"2024-04-26T22:48:30.927373+0000","last_became_active":"2024-04-26T22:47:40.767450+0000","last_became_peered":"2024-04-26T22:47:40.767450+0000","last_unstale":"2024-04-26T22:48:30.927373+0000","last_undegraded":"2024-04-26T22:48:30.927373+0000","last_fullsized":"2024-04-26T22:48:30.927373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:41.481981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992902+0000","last_change":"2024-04-26T22:47:41.774308+0000","last_active":"2024-04-26T22:48:30.992902+0000","last_peered":"2024-04-26T22:48:30.992902+0000","last_clean":"2024-04-26T22:48:30.992902+0000","last_became_active":"2024-04-26T22:47:41.773713+0000","last_became_peered":"2024-04-26T22:47:41.773713+0000","last_unstale":"2024-04-26T22:48:30.992902+0000","last_undegraded":"2024-04-26T22:48:30.992902+0000","last_fullsized":"2024-04-26T22:48:30.992902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:11.186258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919818+0000","last_change":"2024-04-26T22:47:40.768761+0000","last_active":"2024-04-26T22:48:30.919818+0000","last_peered":"2024-04-26T22:48:30.919818+0000","last_clean":"2024-04-26T22:48:30.919818+0000","last_became_active":"2024-04-26T22:47:40.768434+0000","last_became_peered":"2024-04-26T22:47:40.768434+0000","last_unstale":"2024-04-26T22:48:30.919818+0000","last_undegraded":"2024-04-26T22:48:30.919818+0000","last_fullsized":"2024-04-26T22:48:30.919818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:01.631990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927510+0000","last_change":"2024-04-26T22:47:41.778214+0000","last_active":"2024-04-26T22:48:30.927510+0000","last_peered":"2024-04-26T22:48:30.927510+0000","last_clean":"2024-04-26T22:48:30.927510+0000","last_became_active":"2024-04-26T22:47:41.775889+0000","last_became_peered":"2024-04-26T22:47:41.775889+0000","last_unstale":"2024-04-26T22:48:30.927510+0000","last_undegraded":"2024-04-26T22:48:30.927510+0000","last_fullsized":"2024-04-26T22:48:30.927510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:24.654113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:53.898756+0000","last_change":"2024-04-26T22:47:40.772366+0000","last_active":"2024-04-26T22:48:53.898756+0000","last_peered":"2024-04-26T22:48:53.898756+0000","last_clean":"2024-04-26T22:48:53.898756+0000","last_became_active":"2024-04-26T22:47:40.772035+0000","last_became_peered":"2024-04-26T22:47:40.772035+0000","last_unstale":"2024-04-26T22:48:53.898756+0000","last_undegraded":"2024-04-26T22:48:53.898756+0000","last_fullsized":"2024-04-26T22:48:53.898756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:54.429132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927518+0000","last_change":"2024-04-26T22:47:41.777074+0000","last_active":"2024-04-26T22:48:30.927518+0000","last_peered":"2024-04-26T22:48:30.927518+0000","last_clean":"2024-04-26T22:48:30.927518+0000","last_became_active":"2024-04-26T22:47:41.776225+0000","last_became_peered":"2024-04-26T22:47:41.776225+0000","last_unstale":"2024-04-26T22:48:30.927518+0000","last_undegraded":"2024-04-26T22:48:30.927518+0000","last_fullsized":"2024-04-26T22:48:30.927518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:06:59.801103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924013+0000","last_change":"2024-04-26T22:47:40.776174+0000","last_active":"2024-04-26T22:48:30.924013+0000","last_peered":"2024-04-26T22:48:30.924013+0000","last_clean":"2024-04-26T22:48:30.924013+0000","last_became_active":"2024-04-26T22:47:40.775866+0000","last_became_peered":"2024-04-26T22:47:40.775866+0000","last_unstale":"2024-04-26T22:48:30.924013+0000","last_undegraded":"2024-04-26T22:48:30.924013+0000","last_fullsized":"2024-04-26T22:48:30.924013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:01.306261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927142+0000","last_change":"2024-04-26T22:47:41.777225+0000","last_active":"2024-04-26T22:48:30.927142+0000","last_peered":"2024-04-26T22:48:30.927142+0000","last_clean":"2024-04-26T22:48:30.927142+0000","last_became_active":"2024-04-26T22:47:41.776814+0000","last_became_peered":"2024-04-26T22:47:41.776814+0000","last_unstale":"2024-04-26T22:48:30.927142+0000","last_undegraded":"2024-04-26T22:48:30.927142+0000","last_fullsized":"2024-04-26T22:48:30.927142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:30.254893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993963+0000","last_change":"2024-04-26T22:47:40.781678+0000","last_active":"2024-04-26T22:48:30.993963+0000","last_peered":"2024-04-26T22:48:30.993963+0000","last_clean":"2024-04-26T22:48:30.993963+0000","last_became_active":"2024-04-26T22:47:40.781156+0000","last_became_peered":"2024-04-26T22:47:40.781156+0000","last_unstale":"2024-04-26T22:48:30.993963+0000","last_undegraded":"2024-04-26T22:48:30.993963+0000","last_fullsized":"2024-04-26T22:48:30.993963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:53.353768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.928064+0000","last_change":"2024-04-26T22:47:41.780116+0000","last_active":"2024-04-26T22:48:30.928064+0000","last_peered":"2024-04-26T22:48:30.928064+0000","last_clean":"2024-04-26T22:48:30.928064+0000","last_became_active":"2024-04-26T22:47:41.777443+0000","last_became_peered":"2024-04-26T22:47:41.777443+0000","last_unstale":"2024-04-26T22:48:30.928064+0000","last_undegraded":"2024-04-26T22:48:30.928064+0000","last_fullsized":"2024-04-26T22:48:30.928064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:22.948100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921704+0000","last_change":"2024-04-26T22:47:41.784618+0000","last_active":"2024-04-26T22:48:30.921704+0000","last_peered":"2024-04-26T22:48:30.921704+0000","last_clean":"2024-04-26T22:48:30.921704+0000","last_became_active":"2024-04-26T22:47:41.784269+0000","last_became_peered":"2024-04-26T22:47:41.784269+0000","last_unstale":"2024-04-26T22:48:30.921704+0000","last_undegraded":"2024-04-26T22:48:30.921704+0000","last_fullsized":"2024-04-26T22:48:30.921704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:26:08.725744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927636+0000","last_change":"2024-04-26T22:47:41.786063+0000","last_active":"2024-04-26T22:48:30.927636+0000","last_peered":"2024-04-26T22:48:30.927636+0000","last_clean":"2024-04-26T22:48:30.927636+0000","last_became_active":"2024-04-26T22:47:41.785657+0000","last_became_peered":"2024-04-26T22:47:41.785657+0000","last_unstale":"2024-04-26T22:48:30.927636+0000","last_undegraded":"2024-04-26T22:48:30.927636+0000","last_fullsized":"2024-04-26T22:48:30.927636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:14.289477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927261+0000","last_change":"2024-04-26T22:47:41.788788+0000","last_active":"2024-04-26T22:48:30.927261+0000","last_peered":"2024-04-26T22:48:30.927261+0000","last_clean":"2024-04-26T22:48:30.927261+0000","last_became_active":"2024-04-26T22:47:41.788434+0000","last_became_peered":"2024-04-26T22:47:41.788434+0000","last_unstale":"2024-04-26T22:48:30.927261+0000","last_undegraded":"2024-04-26T22:48:30.927261+0000","last_fullsized":"2024-04-26T22:48:30.927261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:05.056380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919779+0000","last_change":"2024-04-26T22:47:41.780435+0000","last_active":"2024-04-26T22:48:30.919779+0000","last_peered":"2024-04-26T22:48:30.919779+0000","last_clean":"2024-04-26T22:48:30.919779+0000","last_became_active":"2024-04-26T22:47:41.780126+0000","last_became_peered":"2024-04-26T22:47:41.780126+0000","last_unstale":"2024-04-26T22:48:30.919779+0000","last_undegraded":"2024-04-26T22:48:30.919779+0000","last_fullsized":"2024-04-26T22:48:30.919779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:20.061814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993984+0000","last_change":"2024-04-26T22:47:41.775797+0000","last_active":"2024-04-26T22:48:30.993984+0000","last_peered":"2024-04-26T22:48:30.993984+0000","last_clean":"2024-04-26T22:48:30.993984+0000","last_became_active":"2024-04-26T22:47:41.775286+0000","last_became_peered":"2024-04-26T22:47:41.775286+0000","last_unstale":"2024-04-26T22:48:30.993984+0000","last_undegraded":"2024-04-26T22:48:30.993984+0000","last_fullsized":"2024-04-26T22:48:30.993984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:45.290373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921900+0000","last_change":"2024-04-26T22:47:41.784714+0000","last_active":"2024-04-26T22:48:30.921900+0000","last_peered":"2024-04-26T22:48:30.921900+0000","last_clean":"2024-04-26T22:48:30.921900+0000","last_became_active":"2024-04-26T22:47:41.783233+0000","last_became_peered":"2024-04-26T22:47:41.783233+0000","last_unstale":"2024-04-26T22:48:30.921900+0000","last_undegraded":"2024-04-26T22:48:30.921900+0000","last_fullsized":"2024-04-26T22:48:30.921900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:34.350512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927660+0000","last_change":"2024-04-26T22:47:41.796781+0000","last_active":"2024-04-26T22:48:30.927660+0000","last_peered":"2024-04-26T22:48:30.927660+0000","last_clean":"2024-04-26T22:48:30.927660+0000","last_became_active":"2024-04-26T22:47:41.796326+0000","last_became_peered":"2024-04-26T22:47:41.796326+0000","last_unstale":"2024-04-26T22:48:30.927660+0000","last_undegraded":"2024-04-26T22:48:30.927660+0000","last_fullsized":"2024-04-26T22:48:30.927660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:30:59.924808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993232+0000","last_change":"2024-04-26T22:47:41.785149+0000","last_active":"2024-04-26T22:48:30.993232+0000","last_peered":"2024-04-26T22:48:30.993232+0000","last_clean":"2024-04-26T22:48:30.993232+0000","last_became_active":"2024-04-26T22:47:41.784807+0000","last_became_peered":"2024-04-26T22:47:41.784807+0000","last_unstale":"2024-04-26T22:48:30.993232+0000","last_undegraded":"2024-04-26T22:48:30.993232+0000","last_fullsized":"2024-04-26T22:48:30.993232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:45.521731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923924+0000","last_change":"2024-04-26T22:47:41.768789+0000","last_active":"2024-04-26T22:48:30.923924+0000","last_peered":"2024-04-26T22:48:30.923924+0000","last_clean":"2024-04-26T22:48:30.923924+0000","last_became_active":"2024-04-26T22:47:41.768466+0000","last_became_peered":"2024-04-26T22:47:41.768466+0000","last_unstale":"2024-04-26T22:48:30.923924+0000","last_undegraded":"2024-04-26T22:48:30.923924+0000","last_fullsized":"2024-04-26T22:48:30.923924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:15.798301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992885+0000","last_change":"2024-04-26T22:47:41.781867+0000","last_active":"2024-04-26T22:48:30.992885+0000","last_peered":"2024-04-26T22:48:30.992885+0000","last_clean":"2024-04-26T22:48:30.992885+0000","last_became_active":"2024-04-26T22:47:41.780648+0000","last_became_peered":"2024-04-26T22:47:41.780648+0000","last_unstale":"2024-04-26T22:48:30.992885+0000","last_undegraded":"2024-04-26T22:48:30.992885+0000","last_fullsized":"2024-04-26T22:48:30.992885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:25.393892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923684+0000","last_change":"2024-04-26T22:47:41.774622+0000","last_active":"2024-04-26T22:48:30.923684+0000","last_peered":"2024-04-26T22:48:30.923684+0000","last_clean":"2024-04-26T22:48:30.923684+0000","last_became_active":"2024-04-26T22:47:41.774262+0000","last_became_peered":"2024-04-26T22:47:41.774262+0000","last_unstale":"2024-04-26T22:48:30.923684+0000","last_undegraded":"2024-04-26T22:48:30.923684+0000","last_fullsized":"2024-04-26T22:48:30.923684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:32.571497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993773+0000","last_change":"2024-04-26T22:47:41.768981+0000","last_active":"2024-04-26T22:48:30.993773+0000","last_peered":"2024-04-26T22:48:30.993773+0000","last_clean":"2024-04-26T22:48:30.993773+0000","last_became_active":"2024-04-26T22:47:41.768515+0000","last_became_peered":"2024-04-26T22:47:41.768515+0000","last_unstale":"2024-04-26T22:48:30.993773+0000","last_undegraded":"2024-04-26T22:48:30.993773+0000","last_fullsized":"2024-04-26T22:48:30.993773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:43:47.680248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921751+0000","last_change":"2024-04-26T22:47:41.788415+0000","last_active":"2024-04-26T22:48:30.921751+0000","last_peered":"2024-04-26T22:48:30.921751+0000","last_clean":"2024-04-26T22:48:30.921751+0000","last_became_active":"2024-04-26T22:47:41.788076+0000","last_became_peered":"2024-04-26T22:47:41.788076+0000","last_unstale":"2024-04-26T22:48:30.921751+0000","last_undegraded":"2024-04-26T22:48:30.921751+0000","last_fullsized":"2024-04-26T22:48:30.921751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:51:54.658275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":288005,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57602,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":734,"num_read_kb":429,"num_write":232,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":172806,"data_compressed":5538,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":457141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":1098546,"data_compressed":38462,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":456658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":457014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1049554,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":456375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":507686,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":458105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":508297,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":456200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920834,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":456627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":1048841,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":52150,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":52761,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":52525,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T22:49:01.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T22:49:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:01.696 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:01 smithi001 ceph-mon[29435]: pgmap v332: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:01.696 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:01 smithi001 ceph-mon[29435]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:49:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:01 smithi186 ceph-mon[32774]: pgmap v332: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:01 smithi186 ceph-mon[32774]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:49:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:01 smithi038 ceph-mon[32433]: pgmap v332: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:01 smithi038 ceph-mon[32433]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:49:03.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 -- 172.21.15.1:0/917878614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 msgr2=0x7fbfe4102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/917878614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4102990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40099c0 tx=0x7fbfd4009cd0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 -- 172.21.15.1:0/917878614 shutdown_connections 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/917878614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe4107640 0x7fbfe410f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/917878614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4102990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/917878614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfe4102f60 0x7fbfe4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 -- 172.21.15.1:0/917878614 >> 172.21.15.1:0/917878614 conn(0x7fbfe40792d0 msgr2=0x7fbfe40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 -- 172.21.15.1:0/917878614 shutdown_connections 2024-04-26T22:49:03.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.221+0000 7fbfea9bf700 1 -- 172.21.15.1:0/917878614 wait complete. 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 Processor -- start 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 -- start start 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfe4102f60 0x7fbfe4104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe4107640 0x7fbfe4104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe4112550 con 0x7fbfe41025b0 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe41123d0 con 0x7fbfe4102f60 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfea9bf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe41126d0 con 0x7fbfe4107640 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4104000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46748/0 (socket says 172.21.15.1:46748) 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfe3fff700 1 -- 172.21.15.1:0/1814294600 learned_addr learned my addr 172.21.15.1:0/1814294600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.222+0000 7fbfe37fe700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfe4102f60 0x7fbfe4104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe8f5c700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe4107640 0x7fbfe4104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe3fff700 1 -- 172.21.15.1:0/1814294600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe4107640 msgr2=0x7fbfe4104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe3fff700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe4107640 0x7fbfe4104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe3fff700 1 -- 172.21.15.1:0/1814294600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfe4102f60 msgr2=0x7fbfe4104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe3fff700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfe4102f60 0x7fbfe4104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe3fff700 1 -- 172.21.15.1:0/1814294600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe41a80c0 con 0x7fbfe41025b0 2024-04-26T22:49:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe3fff700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4104000 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400b580 tx=0x7fbfd40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4027070 con 0x7fbfe41025b0 2024-04-26T22:49:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfd40040a0 con 0x7fbfe41025b0 2024-04-26T22:49:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe41a8350 con 0x7fbfe41025b0 2024-04-26T22:49:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400f460 con 0x7fbfe41025b0 2024-04-26T22:49:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.223+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe41a87a0 con 0x7fbfe41025b0 2024-04-26T22:49:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.224+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe406a110 con 0x7fbfe41025b0 2024-04-26T22:49:03.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.225+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbfd402a070 con 0x7fbfe41025b0 2024-04-26T22:49:03.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.226+0000 7fbfe17fa700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbfcc0a3cd0 0x7fbfcc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:03.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.226+0000 7fbfe37fe700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbfcc0a3cd0 0x7fbfcc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:03.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.226+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fbfd40d15e0 con 0x7fbfe41025b0 2024-04-26T22:49:03.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.227+0000 7fbfe37fe700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbfcc0a3cd0 0x7fbfcc0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbfe41022b0 tx=0x7fbfd800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:03.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.231+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbfd405f060 con 0x7fbfe41025b0 2024-04-26T22:49:03.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.432+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbfe4063860 con 0x7fbfcc0a3cd0 2024-04-26T22:49:03.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.440+0000 7fbfe17fa700 1 -- 172.21.15.1:0/1814294600 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258983 (secure 0 0 0) 0x7fbfe4063860 con 0x7fbfcc0a3cd0 2024-04-26T22:49:03.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.442+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbfcc0a3cd0 msgr2=0x7fbfcc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.442+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbfcc0a3cd0 0x7fbfcc0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbfe41022b0 tx=0x7fbfd800a300 comp rx=0 tx=0).stop 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.442+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 msgr2=0x7fbfe4104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.442+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4104000 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400b580 tx=0x7fbfd40047b0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 shutdown_connections 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbfcc0a3cd0 0x7fbfcc0a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe4107640 0x7fbfe4104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfe41025b0 0x7fbfe4104000 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 --2- 172.21.15.1:0/1814294600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfe4102f60 0x7fbfe4104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:03.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 >> 172.21.15.1:0/1814294600 conn(0x7fbfe40792d0 msgr2=0x7fbfe406beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:03.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 shutdown_connections 2024-04-26T22:49:03.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:03.443+0000 7fbfea9bf700 1 -- 172.21.15.1:0/1814294600 wait complete. 2024-04-26T22:49:03.446 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T22:49:03.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:03 smithi186 ceph-mon[32774]: pgmap v333: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:03.805 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:03 smithi001 ceph-mon[29435]: pgmap v333: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:03 smithi038 ceph-mon[32433]: pgmap v333: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-26T22:49:01.770027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648204,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":900,"num_read_kb":563,"num_write":415,"num_write_kb":3061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360904,"kb_used_data":11676,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580920,"statfs":{"total":1151940427776,"available":1151570862080,"internally_reserved":0,"allocated":11956224,"data_stored":7411044,"data_compressed":115386,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":75,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922625+0000","last_change":"2024-04-26T22:47:41.777587+0000","last_active":"2024-04-26T22:48:30.922625+0000","last_peered":"2024-04-26T22:48:30.922625+0000","last_clean":"2024-04-26T22:48:30.922625+0000","last_became_active":"2024-04-26T22:47:41.777165+0000","last_became_peered":"2024-04-26T22:47:41.777165+0000","last_unstale":"2024-04-26T22:48:30.922625+0000","last_undegraded":"2024-04-26T22:48:30.922625+0000","last_fullsized":"2024-04-26T22:48:30.922625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:45.131599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919933+0000","last_change":"2024-04-26T22:47:41.776888+0000","last_active":"2024-04-26T22:48:30.919933+0000","last_peered":"2024-04-26T22:48:30.919933+0000","last_clean":"2024-04-26T22:48:30.919933+0000","last_became_active":"2024-04-26T22:47:41.776580+0000","last_became_peered":"2024-04-26T22:47:41.776580+0000","last_unstale":"2024-04-26T22:48:30.919933+0000","last_undegraded":"2024-04-26T22:48:30.919933+0000","last_fullsized":"2024-04-26T22:48:30.919933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:49.263390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919709+0000","last_change":"2024-04-26T22:47:41.784374+0000","last_active":"2024-04-26T22:48:30.919709+0000","last_peered":"2024-04-26T22:48:30.919709+0000","last_clean":"2024-04-26T22:48:30.919709+0000","last_became_active":"2024-04-26T22:47:41.783119+0000","last_became_peered":"2024-04-26T22:47:41.783119+0000","last_unstale":"2024-04-26T22:48:30.919709+0000","last_undegraded":"2024-04-26T22:48:30.919709+0000","last_fullsized":"2024-04-26T22:48:30.919709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:18.599515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924713+0000","last_change":"2024-04-26T22:47:41.782485+0000","last_active":"2024-04-26T22:48:30.924713+0000","last_peered":"2024-04-26T22:48:30.924713+0000","last_clean":"2024-04-26T22:48:30.924713+0000","last_became_active":"2024-04-26T22:47:41.782118+0000","last_became_peered":"2024-04-26T22:47:41.782118+0000","last_unstale":"2024-04-26T22:48:30.924713+0000","last_undegraded":"2024-04-26T22:48:30.924713+0000","last_fullsized":"2024-04-26T22:48:30.924713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:39.953297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919757+0000","last_change":"2024-04-26T22:47:41.776281+0000","last_active":"2024-04-26T22:48:30.919757+0000","last_peered":"2024-04-26T22:48:30.919757+0000","last_clean":"2024-04-26T22:48:30.919757+0000","last_became_active":"2024-04-26T22:47:41.775838+0000","last_became_peered":"2024-04-26T22:47:41.775838+0000","last_unstale":"2024-04-26T22:48:30.919757+0000","last_undegraded":"2024-04-26T22:48:30.919757+0000","last_fullsized":"2024-04-26T22:48:30.919757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:04.702698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993201+0000","last_change":"2024-04-26T22:47:41.785568+0000","last_active":"2024-04-26T22:48:30.993201+0000","last_peered":"2024-04-26T22:48:30.993201+0000","last_clean":"2024-04-26T22:48:30.993201+0000","last_became_active":"2024-04-26T22:47:41.785188+0000","last_became_peered":"2024-04-26T22:47:41.785188+0000","last_unstale":"2024-04-26T22:48:30.993201+0000","last_undegraded":"2024-04-26T22:48:30.993201+0000","last_fullsized":"2024-04-26T22:48:30.993201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:21.928641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927020+0000","last_change":"2024-04-26T22:47:41.785722+0000","last_active":"2024-04-26T22:48:30.927020+0000","last_peered":"2024-04-26T22:48:30.927020+0000","last_clean":"2024-04-26T22:48:30.927020+0000","last_became_active":"2024-04-26T22:47:41.784935+0000","last_became_peered":"2024-04-26T22:47:41.784935+0000","last_unstale":"2024-04-26T22:48:30.927020+0000","last_undegraded":"2024-04-26T22:48:30.927020+0000","last_fullsized":"2024-04-26T22:48:30.927020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:24.047598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.994932+0000","last_change":"2024-04-26T22:48:30.994932+0000","last_active":"2024-04-26T22:48:30.994932+0000","last_peered":"2024-04-26T22:48:30.994932+0000","last_clean":"2024-04-26T22:48:30.994932+0000","last_became_active":"2024-04-26T22:48:30.994471+0000","last_became_peered":"2024-04-26T22:48:30.994471+0000","last_unstale":"2024-04-26T22:48:30.994932+0000","last_undegraded":"2024-04-26T22:48:30.994932+0000","last_fullsized":"2024-04-26T22:48:30.994932+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:13:08.678628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924764+0000","last_change":"2024-04-26T22:47:41.775207+0000","last_active":"2024-04-26T22:48:30.924764+0000","last_peered":"2024-04-26T22:48:30.924764+0000","last_clean":"2024-04-26T22:48:30.924764+0000","last_became_active":"2024-04-26T22:47:41.774834+0000","last_became_peered":"2024-04-26T22:47:41.774834+0000","last_unstale":"2024-04-26T22:48:30.924764+0000","last_undegraded":"2024-04-26T22:48:30.924764+0000","last_fullsized":"2024-04-26T22:48:30.924764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:25.913385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993516+0000","last_change":"2024-04-26T22:47:41.785894+0000","last_active":"2024-04-26T22:48:30.993516+0000","last_peered":"2024-04-26T22:48:30.993516+0000","last_clean":"2024-04-26T22:48:30.993516+0000","last_became_active":"2024-04-26T22:47:41.785213+0000","last_became_peered":"2024-04-26T22:47:41.785213+0000","last_unstale":"2024-04-26T22:48:30.993516+0000","last_undegraded":"2024-04-26T22:48:30.993516+0000","last_fullsized":"2024-04-26T22:48:30.993516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:44.927776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927464+0000","last_change":"2024-04-26T22:47:41.781532+0000","last_active":"2024-04-26T22:48:30.927464+0000","last_peered":"2024-04-26T22:48:30.927464+0000","last_clean":"2024-04-26T22:48:30.927464+0000","last_became_active":"2024-04-26T22:47:41.781111+0000","last_became_peered":"2024-04-26T22:47:41.781111+0000","last_unstale":"2024-04-26T22:48:30.927464+0000","last_undegraded":"2024-04-26T22:48:30.927464+0000","last_fullsized":"2024-04-26T22:48:30.927464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:54.930233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924644+0000","last_change":"2024-04-26T22:47:41.782289+0000","last_active":"2024-04-26T22:48:30.924644+0000","last_peered":"2024-04-26T22:48:30.924644+0000","last_clean":"2024-04-26T22:48:30.924644+0000","last_became_active":"2024-04-26T22:47:41.781922+0000","last_became_peered":"2024-04-26T22:47:41.781922+0000","last_unstale":"2024-04-26T22:48:30.924644+0000","last_undegraded":"2024-04-26T22:48:30.924644+0000","last_fullsized":"2024-04-26T22:48:30.924644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:30.146485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993630+0000","last_change":"2024-04-26T22:47:41.770970+0000","last_active":"2024-04-26T22:48:30.993630+0000","last_peered":"2024-04-26T22:48:30.993630+0000","last_clean":"2024-04-26T22:48:30.993630+0000","last_became_active":"2024-04-26T22:47:41.770502+0000","last_became_peered":"2024-04-26T22:47:41.770502+0000","last_unstale":"2024-04-26T22:48:30.993630+0000","last_undegraded":"2024-04-26T22:48:30.993630+0000","last_fullsized":"2024-04-26T22:48:30.993630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.988613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919582+0000","last_change":"2024-04-26T22:47:41.779941+0000","last_active":"2024-04-26T22:48:30.919582+0000","last_peered":"2024-04-26T22:48:30.919582+0000","last_clean":"2024-04-26T22:48:30.919582+0000","last_became_active":"2024-04-26T22:47:41.779592+0000","last_became_peered":"2024-04-26T22:47:41.779592+0000","last_unstale":"2024-04-26T22:48:30.919582+0000","last_undegraded":"2024-04-26T22:48:30.919582+0000","last_fullsized":"2024-04-26T22:48:30.919582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:58.628553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.995248+0000","last_change":"2024-04-26T22:48:30.995248+0000","last_active":"2024-04-26T22:48:30.995248+0000","last_peered":"2024-04-26T22:48:30.995248+0000","last_clean":"2024-04-26T22:48:30.995248+0000","last_became_active":"2024-04-26T22:48:30.994823+0000","last_became_peered":"2024-04-26T22:48:30.994823+0000","last_unstale":"2024-04-26T22:48:30.995248+0000","last_undegraded":"2024-04-26T22:48:30.995248+0000","last_fullsized":"2024-04-26T22:48:30.995248+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:51:12.855303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923528+0000","last_change":"2024-04-26T22:47:41.771277+0000","last_active":"2024-04-26T22:48:30.923528+0000","last_peered":"2024-04-26T22:48:30.923528+0000","last_clean":"2024-04-26T22:48:30.923528+0000","last_became_active":"2024-04-26T22:47:41.770933+0000","last_became_peered":"2024-04-26T22:47:41.770933+0000","last_unstale":"2024-04-26T22:48:30.923528+0000","last_undegraded":"2024-04-26T22:48:30.923528+0000","last_fullsized":"2024-04-26T22:48:30.923528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:15.045678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922368+0000","last_change":"2024-04-26T22:47:41.778656+0000","last_active":"2024-04-26T22:48:30.922368+0000","last_peered":"2024-04-26T22:48:30.922368+0000","last_clean":"2024-04-26T22:48:30.922368+0000","last_became_active":"2024-04-26T22:47:41.778282+0000","last_became_peered":"2024-04-26T22:47:41.778282+0000","last_unstale":"2024-04-26T22:48:30.922368+0000","last_undegraded":"2024-04-26T22:48:30.922368+0000","last_fullsized":"2024-04-26T22:48:30.922368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:12.987690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922065+0000","last_change":"2024-04-26T22:47:41.776965+0000","last_active":"2024-04-26T22:48:30.922065+0000","last_peered":"2024-04-26T22:48:30.922065+0000","last_clean":"2024-04-26T22:48:30.922065+0000","last_became_active":"2024-04-26T22:47:41.776625+0000","last_became_peered":"2024-04-26T22:47:41.776625+0000","last_unstale":"2024-04-26T22:48:30.922065+0000","last_undegraded":"2024-04-26T22:48:30.922065+0000","last_fullsized":"2024-04-26T22:48:30.922065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:38.601785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927130+0000","last_change":"2024-04-26T22:47:41.796742+0000","last_active":"2024-04-26T22:48:30.927130+0000","last_peered":"2024-04-26T22:48:30.927130+0000","last_clean":"2024-04-26T22:48:30.927130+0000","last_became_active":"2024-04-26T22:47:41.796362+0000","last_became_peered":"2024-04-26T22:47:41.796362+0000","last_unstale":"2024-04-26T22:48:30.927130+0000","last_undegraded":"2024-04-26T22:48:30.927130+0000","last_fullsized":"2024-04-26T22:48:30.927130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:01:55.092075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":106,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:58.911166+0000","last_change":"2024-04-26T22:47:40.771619+0000","last_active":"2024-04-26T22:48:58.911166+0000","last_peered":"2024-04-26T22:48:58.911166+0000","last_clean":"2024-04-26T22:48:58.911166+0000","last_became_active":"2024-04-26T22:47:40.771262+0000","last_became_peered":"2024-04-26T22:47:40.771262+0000","last_unstale":"2024-04-26T22:48:58.911166+0000","last_undegraded":"2024-04-26T22:48:58.911166+0000","last_fullsized":"2024-04-26T22:48:58.911166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:19.528487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993392+0000","last_change":"2024-04-26T22:47:41.784162+0000","last_active":"2024-04-26T22:48:30.993392+0000","last_peered":"2024-04-26T22:48:30.993392+0000","last_clean":"2024-04-26T22:48:30.993392+0000","last_became_active":"2024-04-26T22:47:41.783821+0000","last_became_peered":"2024-04-26T22:47:41.783821+0000","last_unstale":"2024-04-26T22:48:30.993392+0000","last_undegraded":"2024-04-26T22:48:30.993392+0000","last_fullsized":"2024-04-26T22:48:30.993392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:17:04.237927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993480+0000","last_change":"2024-04-26T22:47:40.779858+0000","last_active":"2024-04-26T22:48:30.993480+0000","last_peered":"2024-04-26T22:48:30.993480+0000","last_clean":"2024-04-26T22:48:30.993480+0000","last_became_active":"2024-04-26T22:47:40.779550+0000","last_became_peered":"2024-04-26T22:47:40.779550+0000","last_unstale":"2024-04-26T22:48:30.993480+0000","last_undegraded":"2024-04-26T22:48:30.993480+0000","last_fullsized":"2024-04-26T22:48:30.993480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:40:54.056426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919942+0000","last_change":"2024-04-26T22:47:41.769865+0000","last_active":"2024-04-26T22:48:30.919942+0000","last_peered":"2024-04-26T22:48:30.919942+0000","last_clean":"2024-04-26T22:48:30.919942+0000","last_became_active":"2024-04-26T22:47:41.769545+0000","last_became_peered":"2024-04-26T22:47:41.769545+0000","last_unstale":"2024-04-26T22:48:30.919942+0000","last_undegraded":"2024-04-26T22:48:30.919942+0000","last_fullsized":"2024-04-26T22:48:30.919942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:39.715241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920239+0000","last_change":"2024-04-26T22:47:40.771234+0000","last_active":"2024-04-26T22:48:30.920239+0000","last_peered":"2024-04-26T22:48:30.920239+0000","last_clean":"2024-04-26T22:48:30.920239+0000","last_became_active":"2024-04-26T22:47:40.770876+0000","last_became_peered":"2024-04-26T22:47:40.770876+0000","last_unstale":"2024-04-26T22:48:30.920239+0000","last_undegraded":"2024-04-26T22:48:30.920239+0000","last_fullsized":"2024-04-26T22:48:30.920239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:50:41.272029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923627+0000","last_change":"2024-04-26T22:47:41.770470+0000","last_active":"2024-04-26T22:48:30.923627+0000","last_peered":"2024-04-26T22:48:30.923627+0000","last_clean":"2024-04-26T22:48:30.923627+0000","last_became_active":"2024-04-26T22:47:41.767318+0000","last_became_peered":"2024-04-26T22:47:41.767318+0000","last_unstale":"2024-04-26T22:48:30.923627+0000","last_undegraded":"2024-04-26T22:48:30.923627+0000","last_fullsized":"2024-04-26T22:48:30.923627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:45.324118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920119+0000","last_change":"2024-04-26T22:47:40.775811+0000","last_active":"2024-04-26T22:48:30.920119+0000","last_peered":"2024-04-26T22:48:30.920119+0000","last_clean":"2024-04-26T22:48:30.920119+0000","last_became_active":"2024-04-26T22:47:40.775349+0000","last_became_peered":"2024-04-26T22:47:40.775349+0000","last_unstale":"2024-04-26T22:48:30.920119+0000","last_undegraded":"2024-04-26T22:48:30.920119+0000","last_fullsized":"2024-04-26T22:48:30.920119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.926983+0000","last_change":"2024-04-26T22:48:30.926983+0000","last_active":"2024-04-26T22:48:30.926983+0000","last_peered":"2024-04-26T22:48:30.926983+0000","last_clean":"2024-04-26T22:48:30.926983+0000","last_became_active":"2024-04-26T22:48:30.926588+0000","last_became_peered":"2024-04-26T22:48:30.926588+0000","last_unstale":"2024-04-26T22:48:30.926983+0000","last_undegraded":"2024-04-26T22:48:30.926983+0000","last_fullsized":"2024-04-26T22:48:30.926983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:51:09.309076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:58.905785+0000","last_change":"2024-04-26T22:47:40.774855+0000","last_active":"2024-04-26T22:48:58.905785+0000","last_peered":"2024-04-26T22:48:58.905785+0000","last_clean":"2024-04-26T22:48:58.905785+0000","last_became_active":"2024-04-26T22:47:40.774517+0000","last_became_peered":"2024-04-26T22:47:40.774517+0000","last_unstale":"2024-04-26T22:48:58.905785+0000","last_undegraded":"2024-04-26T22:48:58.905785+0000","last_fullsized":"2024-04-26T22:48:58.905785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.811072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919963+0000","last_change":"2024-04-26T22:47:41.781432+0000","last_active":"2024-04-26T22:48:30.919963+0000","last_peered":"2024-04-26T22:48:30.919963+0000","last_clean":"2024-04-26T22:48:30.919963+0000","last_became_active":"2024-04-26T22:47:41.781035+0000","last_became_peered":"2024-04-26T22:47:41.781035+0000","last_unstale":"2024-04-26T22:48:30.919963+0000","last_undegraded":"2024-04-26T22:48:30.919963+0000","last_fullsized":"2024-04-26T22:48:30.919963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:31.730087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993235+0000","last_change":"2024-04-26T22:47:40.770419+0000","last_active":"2024-04-26T22:48:30.993235+0000","last_peered":"2024-04-26T22:48:30.993235+0000","last_clean":"2024-04-26T22:48:30.993235+0000","last_became_active":"2024-04-26T22:47:40.769923+0000","last_became_peered":"2024-04-26T22:47:40.769923+0000","last_unstale":"2024-04-26T22:48:30.993235+0000","last_undegraded":"2024-04-26T22:48:30.993235+0000","last_fullsized":"2024-04-26T22:48:30.993235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:15.618904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924075+0000","last_change":"2024-04-26T22:47:41.778334+0000","last_active":"2024-04-26T22:48:30.924075+0000","last_peered":"2024-04-26T22:48:30.924075+0000","last_clean":"2024-04-26T22:48:30.924075+0000","last_became_active":"2024-04-26T22:47:41.777822+0000","last_became_peered":"2024-04-26T22:47:41.777822+0000","last_unstale":"2024-04-26T22:48:30.924075+0000","last_undegraded":"2024-04-26T22:48:30.924075+0000","last_fullsized":"2024-04-26T22:48:30.924075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:57.828165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993903+0000","last_change":"2024-04-26T22:47:40.779524+0000","last_active":"2024-04-26T22:48:30.993903+0000","last_peered":"2024-04-26T22:48:30.993903+0000","last_clean":"2024-04-26T22:48:30.993903+0000","last_became_active":"2024-04-26T22:47:40.778998+0000","last_became_peered":"2024-04-26T22:47:40.778998+0000","last_unstale":"2024-04-26T22:48:30.993903+0000","last_undegraded":"2024-04-26T22:48:30.993903+0000","last_fullsized":"2024-04-26T22:48:30.993903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:59.979648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992671+0000","last_change":"2024-04-26T22:47:41.768883+0000","last_active":"2024-04-26T22:48:30.992671+0000","last_peered":"2024-04-26T22:48:30.992671+0000","last_clean":"2024-04-26T22:48:30.992671+0000","last_became_active":"2024-04-26T22:47:41.768577+0000","last_became_peered":"2024-04-26T22:47:41.768577+0000","last_unstale":"2024-04-26T22:48:30.992671+0000","last_undegraded":"2024-04-26T22:48:30.992671+0000","last_fullsized":"2024-04-26T22:48:30.992671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:29.267873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923845+0000","last_change":"2024-04-26T22:47:40.774688+0000","last_active":"2024-04-26T22:48:30.923845+0000","last_peered":"2024-04-26T22:48:30.923845+0000","last_clean":"2024-04-26T22:48:30.923845+0000","last_became_active":"2024-04-26T22:47:40.774312+0000","last_became_peered":"2024-04-26T22:47:40.774312+0000","last_unstale":"2024-04-26T22:48:30.923845+0000","last_undegraded":"2024-04-26T22:48:30.923845+0000","last_fullsized":"2024-04-26T22:48:30.923845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:03:34.797965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921763+0000","last_change":"2024-04-26T22:47:41.776948+0000","last_active":"2024-04-26T22:48:30.921763+0000","last_peered":"2024-04-26T22:48:30.921763+0000","last_clean":"2024-04-26T22:48:30.921763+0000","last_became_active":"2024-04-26T22:47:41.776583+0000","last_became_peered":"2024-04-26T22:47:41.776583+0000","last_unstale":"2024-04-26T22:48:30.921763+0000","last_undegraded":"2024-04-26T22:48:30.921763+0000","last_fullsized":"2024-04-26T22:48:30.921763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:44:42.687249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":173,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:49:00.907723+0000","last_change":"2024-04-26T22:47:40.772060+0000","last_active":"2024-04-26T22:49:00.907723+0000","last_peered":"2024-04-26T22:49:00.907723+0000","last_clean":"2024-04-26T22:49:00.907723+0000","last_became_active":"2024-04-26T22:47:40.771690+0000","last_became_peered":"2024-04-26T22:47:40.771690+0000","last_unstale":"2024-04-26T22:49:00.907723+0000","last_undegraded":"2024-04-26T22:49:00.907723+0000","last_fullsized":"2024-04-26T22:49:00.907723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:53.619399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924433+0000","last_change":"2024-04-26T22:47:41.776902+0000","last_active":"2024-04-26T22:48:30.924433+0000","last_peered":"2024-04-26T22:48:30.924433+0000","last_clean":"2024-04-26T22:48:30.924433+0000","last_became_active":"2024-04-26T22:47:41.776475+0000","last_became_peered":"2024-04-26T22:47:41.776475+0000","last_unstale":"2024-04-26T22:48:30.924433+0000","last_undegraded":"2024-04-26T22:48:30.924433+0000","last_fullsized":"2024-04-26T22:48:30.924433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:57:17.674715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927603+0000","last_change":"2024-04-26T22:47:40.775828+0000","last_active":"2024-04-26T22:48:30.927603+0000","last_peered":"2024-04-26T22:48:30.927603+0000","last_clean":"2024-04-26T22:48:30.927603+0000","last_became_active":"2024-04-26T22:47:40.775439+0000","last_became_peered":"2024-04-26T22:47:40.775439+0000","last_unstale":"2024-04-26T22:48:30.927603+0000","last_undegraded":"2024-04-26T22:48:30.927603+0000","last_fullsized":"2024-04-26T22:48:30.927603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:28:21.061843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919938+0000","last_change":"2024-04-26T22:47:41.782731+0000","last_active":"2024-04-26T22:48:30.919938+0000","last_peered":"2024-04-26T22:48:30.919938+0000","last_clean":"2024-04-26T22:48:30.919938+0000","last_became_active":"2024-04-26T22:47:41.781390+0000","last_became_peered":"2024-04-26T22:47:41.781390+0000","last_unstale":"2024-04-26T22:48:30.919938+0000","last_undegraded":"2024-04-26T22:48:30.919938+0000","last_fullsized":"2024-04-26T22:48:30.919938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:15.349252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924169+0000","last_change":"2024-04-26T22:47:40.770488+0000","last_active":"2024-04-26T22:48:30.924169+0000","last_peered":"2024-04-26T22:48:30.924169+0000","last_clean":"2024-04-26T22:48:30.924169+0000","last_became_active":"2024-04-26T22:47:40.770049+0000","last_became_peered":"2024-04-26T22:47:40.770049+0000","last_unstale":"2024-04-26T22:48:30.924169+0000","last_undegraded":"2024-04-26T22:48:30.924169+0000","last_fullsized":"2024-04-26T22:48:30.924169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:03.391917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923853+0000","last_change":"2024-04-26T22:47:41.778433+0000","last_active":"2024-04-26T22:48:30.923853+0000","last_peered":"2024-04-26T22:48:30.923853+0000","last_clean":"2024-04-26T22:48:30.923853+0000","last_became_active":"2024-04-26T22:47:41.777603+0000","last_became_peered":"2024-04-26T22:47:41.777603+0000","last_unstale":"2024-04-26T22:48:30.923853+0000","last_undegraded":"2024-04-26T22:48:30.923853+0000","last_fullsized":"2024-04-26T22:48:30.923853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:02.206291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'27","reported_seq":202,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:49:00.907978+0000","last_change":"2024-04-26T22:47:40.774417+0000","last_active":"2024-04-26T22:49:00.907978+0000","last_peered":"2024-04-26T22:49:00.907978+0000","last_clean":"2024-04-26T22:49:00.907978+0000","last_became_active":"2024-04-26T22:47:40.773932+0000","last_became_peered":"2024-04-26T22:47:40.773932+0000","last_unstale":"2024-04-26T22:49:00.907978+0000","last_undegraded":"2024-04-26T22:49:00.907978+0000","last_fullsized":"2024-04-26T22:49:00.907978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:15.694401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50398,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":156,"num_read_kb":127,"num_write":29,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920014+0000","last_change":"2024-04-26T22:47:41.787466+0000","last_active":"2024-04-26T22:48:30.920014+0000","last_peered":"2024-04-26T22:48:30.920014+0000","last_clean":"2024-04-26T22:48:30.920014+0000","last_became_active":"2024-04-26T22:47:41.787105+0000","last_became_peered":"2024-04-26T22:47:41.787105+0000","last_unstale":"2024-04-26T22:48:30.920014+0000","last_undegraded":"2024-04-26T22:48:30.920014+0000","last_fullsized":"2024-04-26T22:48:30.920014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:48:34.656495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927272+0000","last_change":"2024-04-26T22:47:40.767810+0000","last_active":"2024-04-26T22:48:30.927272+0000","last_peered":"2024-04-26T22:48:30.927272+0000","last_clean":"2024-04-26T22:48:30.927272+0000","last_became_active":"2024-04-26T22:47:40.767508+0000","last_became_peered":"2024-04-26T22:47:40.767508+0000","last_unstale":"2024-04-26T22:48:30.927272+0000","last_undegraded":"2024-04-26T22:48:30.927272+0000","last_fullsized":"2024-04-26T22:48:30.927272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:32.023939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993557+0000","last_change":"2024-04-26T22:47:41.774134+0000","last_active":"2024-04-26T22:48:30.993557+0000","last_peered":"2024-04-26T22:48:30.993557+0000","last_clean":"2024-04-26T22:48:30.993557+0000","last_became_active":"2024-04-26T22:47:41.773613+0000","last_became_peered":"2024-04-26T22:47:41.773613+0000","last_unstale":"2024-04-26T22:48:30.993557+0000","last_undegraded":"2024-04-26T22:48:30.993557+0000","last_fullsized":"2024-04-26T22:48:30.993557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:41:51.317633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923756+0000","last_change":"2024-04-26T22:47:40.776836+0000","last_active":"2024-04-26T22:48:30.923756+0000","last_peered":"2024-04-26T22:48:30.923756+0000","last_clean":"2024-04-26T22:48:30.923756+0000","last_became_active":"2024-04-26T22:47:40.776493+0000","last_became_peered":"2024-04-26T22:47:40.776493+0000","last_unstale":"2024-04-26T22:48:30.923756+0000","last_undegraded":"2024-04-26T22:48:30.923756+0000","last_fullsized":"2024-04-26T22:48:30.923756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:55.695456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.995002+0000","last_change":"2024-04-26T22:48:30.995002+0000","last_active":"2024-04-26T22:48:30.995002+0000","last_peered":"2024-04-26T22:48:30.995002+0000","last_clean":"2024-04-26T22:48:30.995002+0000","last_became_active":"2024-04-26T22:48:30.994655+0000","last_became_peered":"2024-04-26T22:48:30.994655+0000","last_unstale":"2024-04-26T22:48:30.995002+0000","last_undegraded":"2024-04-26T22:48:30.995002+0000","last_fullsized":"2024-04-26T22:48:30.995002+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:00.209207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:57.904130+0000","last_change":"2024-04-26T22:47:40.768984+0000","last_active":"2024-04-26T22:48:57.904130+0000","last_peered":"2024-04-26T22:48:57.904130+0000","last_clean":"2024-04-26T22:48:57.904130+0000","last_became_active":"2024-04-26T22:47:40.768596+0000","last_became_peered":"2024-04-26T22:47:40.768596+0000","last_unstale":"2024-04-26T22:48:57.904130+0000","last_undegraded":"2024-04-26T22:48:57.904130+0000","last_fullsized":"2024-04-26T22:48:57.904130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:44.099790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927308+0000","last_change":"2024-04-26T22:47:41.778100+0000","last_active":"2024-04-26T22:48:30.927308+0000","last_peered":"2024-04-26T22:48:30.927308+0000","last_clean":"2024-04-26T22:48:30.927308+0000","last_became_active":"2024-04-26T22:47:41.777775+0000","last_became_peered":"2024-04-26T22:47:41.777775+0000","last_unstale":"2024-04-26T22:48:30.927308+0000","last_undegraded":"2024-04-26T22:48:30.927308+0000","last_fullsized":"2024-04-26T22:48:30.927308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:52.698089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919693+0000","last_change":"2024-04-26T22:47:40.761904+0000","last_active":"2024-04-26T22:48:30.919693+0000","last_peered":"2024-04-26T22:48:30.919693+0000","last_clean":"2024-04-26T22:48:30.919693+0000","last_became_active":"2024-04-26T22:47:40.761591+0000","last_became_peered":"2024-04-26T22:47:40.761591+0000","last_unstale":"2024-04-26T22:48:30.919693+0000","last_undegraded":"2024-04-26T22:48:30.919693+0000","last_fullsized":"2024-04-26T22:48:30.919693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:55.774669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923569+0000","last_change":"2024-04-26T22:47:41.780703+0000","last_active":"2024-04-26T22:48:30.923569+0000","last_peered":"2024-04-26T22:48:30.923569+0000","last_clean":"2024-04-26T22:48:30.923569+0000","last_became_active":"2024-04-26T22:47:41.775023+0000","last_became_peered":"2024-04-26T22:47:41.775023+0000","last_unstale":"2024-04-26T22:48:30.923569+0000","last_undegraded":"2024-04-26T22:48:30.923569+0000","last_fullsized":"2024-04-26T22:48:30.923569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:09.515514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.933400+0000","last_change":"2024-04-26T22:48:30.932931+0000","last_active":"2024-04-26T22:48:30.933400+0000","last_peered":"2024-04-26T22:48:30.933400+0000","last_clean":"2024-04-26T22:48:30.933400+0000","last_became_active":"2024-04-26T22:48:30.926196+0000","last_became_peered":"2024-04-26T22:48:30.926196+0000","last_unstale":"2024-04-26T22:48:30.933400+0000","last_undegraded":"2024-04-26T22:48:30.933400+0000","last_fullsized":"2024-04-26T22:48:30.933400+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:43:44.026285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922011+0000","last_change":"2024-04-26T22:47:41.781793+0000","last_active":"2024-04-26T22:48:30.922011+0000","last_peered":"2024-04-26T22:48:30.922011+0000","last_clean":"2024-04-26T22:48:30.922011+0000","last_became_active":"2024-04-26T22:47:41.781470+0000","last_became_peered":"2024-04-26T22:47:41.781470+0000","last_unstale":"2024-04-26T22:48:30.922011+0000","last_undegraded":"2024-04-26T22:48:30.922011+0000","last_fullsized":"2024-04-26T22:48:30.922011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:20.600909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921912+0000","last_change":"2024-04-26T22:47:40.767364+0000","last_active":"2024-04-26T22:48:30.921912+0000","last_peered":"2024-04-26T22:48:30.921912+0000","last_clean":"2024-04-26T22:48:30.921912+0000","last_became_active":"2024-04-26T22:47:40.766907+0000","last_became_peered":"2024-04-26T22:47:40.766907+0000","last_unstale":"2024-04-26T22:48:30.921912+0000","last_undegraded":"2024-04-26T22:48:30.921912+0000","last_fullsized":"2024-04-26T22:48:30.921912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:56.408996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993511+0000","last_change":"2024-04-26T22:47:41.784095+0000","last_active":"2024-04-26T22:48:30.993511+0000","last_peered":"2024-04-26T22:48:30.993511+0000","last_clean":"2024-04-26T22:48:30.993511+0000","last_became_active":"2024-04-26T22:47:41.783749+0000","last_became_peered":"2024-04-26T22:47:41.783749+0000","last_unstale":"2024-04-26T22:48:30.993511+0000","last_undegraded":"2024-04-26T22:48:30.993511+0000","last_fullsized":"2024-04-26T22:48:30.993511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:11:26.530223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:55.907717+0000","last_change":"2024-04-26T22:47:40.778988+0000","last_active":"2024-04-26T22:48:55.907717+0000","last_peered":"2024-04-26T22:48:55.907717+0000","last_clean":"2024-04-26T22:48:55.907717+0000","last_became_active":"2024-04-26T22:47:40.778686+0000","last_became_peered":"2024-04-26T22:47:40.778686+0000","last_unstale":"2024-04-26T22:48:55.907717+0000","last_undegraded":"2024-04-26T22:48:55.907717+0000","last_fullsized":"2024-04-26T22:48:55.907717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.525866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993231+0000","last_change":"2024-04-26T22:47:41.787168+0000","last_active":"2024-04-26T22:48:30.993231+0000","last_peered":"2024-04-26T22:48:30.993231+0000","last_clean":"2024-04-26T22:48:30.993231+0000","last_became_active":"2024-04-26T22:47:41.786811+0000","last_became_peered":"2024-04-26T22:47:41.786811+0000","last_unstale":"2024-04-26T22:48:30.993231+0000","last_undegraded":"2024-04-26T22:48:30.993231+0000","last_fullsized":"2024-04-26T22:48:30.993231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:34.578058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.929267+0000","last_change":"2024-04-26T22:48:30.929267+0000","last_active":"2024-04-26T22:48:30.929267+0000","last_peered":"2024-04-26T22:48:30.929267+0000","last_clean":"2024-04-26T22:48:30.929267+0000","last_became_active":"2024-04-26T22:48:30.928927+0000","last_became_peered":"2024-04-26T22:48:30.928927+0000","last_unstale":"2024-04-26T22:48:30.929267+0000","last_undegraded":"2024-04-26T22:48:30.929267+0000","last_fullsized":"2024-04-26T22:48:30.929267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:44.270701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:59.906471+0000","last_change":"2024-04-26T22:47:40.781706+0000","last_active":"2024-04-26T22:48:59.906471+0000","last_peered":"2024-04-26T22:48:59.906471+0000","last_clean":"2024-04-26T22:48:59.906471+0000","last_became_active":"2024-04-26T22:47:40.781014+0000","last_became_peered":"2024-04-26T22:47:40.781014+0000","last_unstale":"2024-04-26T22:48:59.906471+0000","last_undegraded":"2024-04-26T22:48:59.906471+0000","last_fullsized":"2024-04-26T22:48:59.906471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:17:16.523767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993026+0000","last_change":"2024-04-26T22:47:41.772852+0000","last_active":"2024-04-26T22:48:30.993026+0000","last_peered":"2024-04-26T22:48:30.993026+0000","last_clean":"2024-04-26T22:48:30.993026+0000","last_became_active":"2024-04-26T22:47:41.772502+0000","last_became_peered":"2024-04-26T22:47:41.772502+0000","last_unstale":"2024-04-26T22:48:30.993026+0000","last_undegraded":"2024-04-26T22:48:30.993026+0000","last_fullsized":"2024-04-26T22:48:30.993026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:36.870593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924870+0000","last_change":"2024-04-26T22:47:40.768151+0000","last_active":"2024-04-26T22:48:30.924870+0000","last_peered":"2024-04-26T22:48:30.924870+0000","last_clean":"2024-04-26T22:48:30.924870+0000","last_became_active":"2024-04-26T22:47:40.767756+0000","last_became_peered":"2024-04-26T22:47:40.767756+0000","last_unstale":"2024-04-26T22:48:30.924870+0000","last_undegraded":"2024-04-26T22:48:30.924870+0000","last_fullsized":"2024-04-26T22:48:30.924870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:38:09.059145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927569+0000","last_change":"2024-04-26T22:46:24.327741+0000","last_active":"2024-04-26T22:48:30.927569+0000","last_peered":"2024-04-26T22:48:30.927569+0000","last_clean":"2024-04-26T22:48:30.927569+0000","last_became_active":"2024-04-26T22:46:24.327280+0000","last_became_peered":"2024-04-26T22:46:24.327280+0000","last_unstale":"2024-04-26T22:48:30.927569+0000","last_undegraded":"2024-04-26T22:48:30.927569+0000","last_fullsized":"2024-04-26T22:48:30.927569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:42:55.984240+0000","last_clean_scrub_stamp":"2024-04-26T22:42:55.984240+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:47.094131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993665+0000","last_change":"2024-04-26T22:47:41.779195+0000","last_active":"2024-04-26T22:48:30.993665+0000","last_peered":"2024-04-26T22:48:30.993665+0000","last_clean":"2024-04-26T22:48:30.993665+0000","last_became_active":"2024-04-26T22:47:41.778682+0000","last_became_peered":"2024-04-26T22:47:41.778682+0000","last_unstale":"2024-04-26T22:48:30.993665+0000","last_undegraded":"2024-04-26T22:48:30.993665+0000","last_fullsized":"2024-04-26T22:48:30.993665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:37.983340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927918+0000","last_change":"2024-04-26T22:47:40.774817+0000","last_active":"2024-04-26T22:48:30.927918+0000","last_peered":"2024-04-26T22:48:30.927918+0000","last_clean":"2024-04-26T22:48:30.927918+0000","last_became_active":"2024-04-26T22:47:40.772660+0000","last_became_peered":"2024-04-26T22:47:40.772660+0000","last_unstale":"2024-04-26T22:48:30.927918+0000","last_undegraded":"2024-04-26T22:48:30.927918+0000","last_fullsized":"2024-04-26T22:48:30.927918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:30.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.920130+0000","last_change":"2024-04-26T22:47:40.773595+0000","last_active":"2024-04-26T22:48:30.920130+0000","last_peered":"2024-04-26T22:48:30.920130+0000","last_clean":"2024-04-26T22:48:30.920130+0000","last_became_active":"2024-04-26T22:47:40.773199+0000","last_became_peered":"2024-04-26T22:47:40.773199+0000","last_unstale":"2024-04-26T22:48:30.920130+0000","last_undegraded":"2024-04-26T22:48:30.920130+0000","last_fullsized":"2024-04-26T22:48:30.920130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:19:31.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993950+0000","last_change":"2024-04-26T22:47:41.780428+0000","last_active":"2024-04-26T22:48:30.993950+0000","last_peered":"2024-04-26T22:48:30.993950+0000","last_clean":"2024-04-26T22:48:30.993950+0000","last_became_active":"2024-04-26T22:47:41.779468+0000","last_became_peered":"2024-04-26T22:47:41.779468+0000","last_unstale":"2024-04-26T22:48:30.993950+0000","last_undegraded":"2024-04-26T22:48:30.993950+0000","last_fullsized":"2024-04-26T22:48:30.993950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:05.067347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.922003+0000","last_change":"2024-04-26T22:47:40.770624+0000","last_active":"2024-04-26T22:48:30.922003+0000","last_peered":"2024-04-26T22:48:30.922003+0000","last_clean":"2024-04-26T22:48:30.922003+0000","last_became_active":"2024-04-26T22:47:40.770250+0000","last_became_peered":"2024-04-26T22:47:40.770250+0000","last_unstale":"2024-04-26T22:48:30.922003+0000","last_undegraded":"2024-04-26T22:48:30.922003+0000","last_fullsized":"2024-04-26T22:48:30.922003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:16:53.527984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993345+0000","last_change":"2024-04-26T22:47:41.773667+0000","last_active":"2024-04-26T22:48:30.993345+0000","last_peered":"2024-04-26T22:48:30.993345+0000","last_clean":"2024-04-26T22:48:30.993345+0000","last_became_active":"2024-04-26T22:47:41.772876+0000","last_became_peered":"2024-04-26T22:47:41.772876+0000","last_unstale":"2024-04-26T22:48:30.993345+0000","last_undegraded":"2024-04-26T22:48:30.993345+0000","last_fullsized":"2024-04-26T22:48:30.993345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:58.363992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924937+0000","last_change":"2024-04-26T22:47:40.774090+0000","last_active":"2024-04-26T22:48:30.924937+0000","last_peered":"2024-04-26T22:48:30.924937+0000","last_clean":"2024-04-26T22:48:30.924937+0000","last_became_active":"2024-04-26T22:47:40.773543+0000","last_became_peered":"2024-04-26T22:47:40.773543+0000","last_unstale":"2024-04-26T22:48:30.924937+0000","last_undegraded":"2024-04-26T22:48:30.924937+0000","last_fullsized":"2024-04-26T22:48:30.924937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:17:06.907699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919990+0000","last_change":"2024-04-26T22:47:41.781231+0000","last_active":"2024-04-26T22:48:30.919990+0000","last_peered":"2024-04-26T22:48:30.919990+0000","last_clean":"2024-04-26T22:48:30.919990+0000","last_became_active":"2024-04-26T22:47:41.780837+0000","last_became_peered":"2024-04-26T22:47:41.780837+0000","last_unstale":"2024-04-26T22:48:30.919990+0000","last_undegraded":"2024-04-26T22:48:30.919990+0000","last_fullsized":"2024-04-26T22:48:30.919990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:24.491628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.930700+0000","last_change":"2024-04-26T22:48:30.930700+0000","last_active":"2024-04-26T22:48:30.930700+0000","last_peered":"2024-04-26T22:48:30.930700+0000","last_clean":"2024-04-26T22:48:30.930700+0000","last_became_active":"2024-04-26T22:48:30.925999+0000","last_became_peered":"2024-04-26T22:48:30.925999+0000","last_unstale":"2024-04-26T22:48:30.930700+0000","last_undegraded":"2024-04-26T22:48:30.930700+0000","last_fullsized":"2024-04-26T22:48:30.930700+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:46.912066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919715+0000","last_change":"2024-04-26T22:47:41.776683+0000","last_active":"2024-04-26T22:48:30.919715+0000","last_peered":"2024-04-26T22:48:30.919715+0000","last_clean":"2024-04-26T22:48:30.919715+0000","last_became_active":"2024-04-26T22:47:41.776309+0000","last_became_peered":"2024-04-26T22:47:41.776309+0000","last_unstale":"2024-04-26T22:48:30.919715+0000","last_undegraded":"2024-04-26T22:48:30.919715+0000","last_fullsized":"2024-04-26T22:48:30.919715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:45.604270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992790+0000","last_change":"2024-04-26T22:47:40.774392+0000","last_active":"2024-04-26T22:48:30.992790+0000","last_peered":"2024-04-26T22:48:30.992790+0000","last_clean":"2024-04-26T22:48:30.992790+0000","last_became_active":"2024-04-26T22:47:40.773682+0000","last_became_peered":"2024-04-26T22:47:40.773682+0000","last_unstale":"2024-04-26T22:48:30.992790+0000","last_undegraded":"2024-04-26T22:48:30.992790+0000","last_fullsized":"2024-04-26T22:48:30.992790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:06:56.769791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921620+0000","last_change":"2024-04-26T22:47:41.784191+0000","last_active":"2024-04-26T22:48:30.921620+0000","last_peered":"2024-04-26T22:48:30.921620+0000","last_clean":"2024-04-26T22:48:30.921620+0000","last_became_active":"2024-04-26T22:47:41.783884+0000","last_became_peered":"2024-04-26T22:47:41.783884+0000","last_unstale":"2024-04-26T22:48:30.921620+0000","last_undegraded":"2024-04-26T22:48:30.921620+0000","last_fullsized":"2024-04-26T22:48:30.921620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:34.035506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927373+0000","last_change":"2024-04-26T22:47:40.767796+0000","last_active":"2024-04-26T22:48:30.927373+0000","last_peered":"2024-04-26T22:48:30.927373+0000","last_clean":"2024-04-26T22:48:30.927373+0000","last_became_active":"2024-04-26T22:47:40.767450+0000","last_became_peered":"2024-04-26T22:47:40.767450+0000","last_unstale":"2024-04-26T22:48:30.927373+0000","last_undegraded":"2024-04-26T22:48:30.927373+0000","last_fullsized":"2024-04-26T22:48:30.927373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:41.481981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992902+0000","last_change":"2024-04-26T22:47:41.774308+0000","last_active":"2024-04-26T22:48:30.992902+0000","last_peered":"2024-04-26T22:48:30.992902+0000","last_clean":"2024-04-26T22:48:30.992902+0000","last_became_active":"2024-04-26T22:47:41.773713+0000","last_became_peered":"2024-04-26T22:47:41.773713+0000","last_unstale":"2024-04-26T22:48:30.992902+0000","last_undegraded":"2024-04-26T22:48:30.992902+0000","last_fullsized":"2024-04-26T22:48:30.992902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:11.186258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919818+0000","last_change":"2024-04-26T22:47:40.768761+0000","last_active":"2024-04-26T22:48:30.919818+0000","last_peered":"2024-04-26T22:48:30.919818+0000","last_clean":"2024-04-26T22:48:30.919818+0000","last_became_active":"2024-04-26T22:47:40.768434+0000","last_became_peered":"2024-04-26T22:47:40.768434+0000","last_unstale":"2024-04-26T22:48:30.919818+0000","last_undegraded":"2024-04-26T22:48:30.919818+0000","last_fullsized":"2024-04-26T22:48:30.919818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:01.631990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927510+0000","last_change":"2024-04-26T22:47:41.778214+0000","last_active":"2024-04-26T22:48:30.927510+0000","last_peered":"2024-04-26T22:48:30.927510+0000","last_clean":"2024-04-26T22:48:30.927510+0000","last_became_active":"2024-04-26T22:47:41.775889+0000","last_became_peered":"2024-04-26T22:47:41.775889+0000","last_unstale":"2024-04-26T22:48:30.927510+0000","last_undegraded":"2024-04-26T22:48:30.927510+0000","last_fullsized":"2024-04-26T22:48:30.927510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:24.654113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:58.905647+0000","last_change":"2024-04-26T22:47:40.772366+0000","last_active":"2024-04-26T22:48:58.905647+0000","last_peered":"2024-04-26T22:48:58.905647+0000","last_clean":"2024-04-26T22:48:58.905647+0000","last_became_active":"2024-04-26T22:47:40.772035+0000","last_became_peered":"2024-04-26T22:47:40.772035+0000","last_unstale":"2024-04-26T22:48:58.905647+0000","last_undegraded":"2024-04-26T22:48:58.905647+0000","last_fullsized":"2024-04-26T22:48:58.905647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:54.429132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927518+0000","last_change":"2024-04-26T22:47:41.777074+0000","last_active":"2024-04-26T22:48:30.927518+0000","last_peered":"2024-04-26T22:48:30.927518+0000","last_clean":"2024-04-26T22:48:30.927518+0000","last_became_active":"2024-04-26T22:47:41.776225+0000","last_became_peered":"2024-04-26T22:47:41.776225+0000","last_unstale":"2024-04-26T22:48:30.927518+0000","last_undegraded":"2024-04-26T22:48:30.927518+0000","last_fullsized":"2024-04-26T22:48:30.927518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:06:59.801103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.924013+0000","last_change":"2024-04-26T22:47:40.776174+0000","last_active":"2024-04-26T22:48:30.924013+0000","last_peered":"2024-04-26T22:48:30.924013+0000","last_clean":"2024-04-26T22:48:30.924013+0000","last_became_active":"2024-04-26T22:47:40.775866+0000","last_became_peered":"2024-04-26T22:47:40.775866+0000","last_unstale":"2024-04-26T22:48:30.924013+0000","last_undegraded":"2024-04-26T22:48:30.924013+0000","last_fullsized":"2024-04-26T22:48:30.924013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:01.306261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927142+0000","last_change":"2024-04-26T22:47:41.777225+0000","last_active":"2024-04-26T22:48:30.927142+0000","last_peered":"2024-04-26T22:48:30.927142+0000","last_clean":"2024-04-26T22:48:30.927142+0000","last_became_active":"2024-04-26T22:47:41.776814+0000","last_became_peered":"2024-04-26T22:47:41.776814+0000","last_unstale":"2024-04-26T22:48:30.927142+0000","last_undegraded":"2024-04-26T22:48:30.927142+0000","last_fullsized":"2024-04-26T22:48:30.927142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:30.254893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993963+0000","last_change":"2024-04-26T22:47:40.781678+0000","last_active":"2024-04-26T22:48:30.993963+0000","last_peered":"2024-04-26T22:48:30.993963+0000","last_clean":"2024-04-26T22:48:30.993963+0000","last_became_active":"2024-04-26T22:47:40.781156+0000","last_became_peered":"2024-04-26T22:47:40.781156+0000","last_unstale":"2024-04-26T22:48:30.993963+0000","last_undegraded":"2024-04-26T22:48:30.993963+0000","last_fullsized":"2024-04-26T22:48:30.993963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:39.745557+0000","last_clean_scrub_stamp":"2024-04-26T22:47:39.745557+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:53.353768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.928064+0000","last_change":"2024-04-26T22:47:41.780116+0000","last_active":"2024-04-26T22:48:30.928064+0000","last_peered":"2024-04-26T22:48:30.928064+0000","last_clean":"2024-04-26T22:48:30.928064+0000","last_became_active":"2024-04-26T22:47:41.777443+0000","last_became_peered":"2024-04-26T22:47:41.777443+0000","last_unstale":"2024-04-26T22:48:30.928064+0000","last_undegraded":"2024-04-26T22:48:30.928064+0000","last_fullsized":"2024-04-26T22:48:30.928064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:22.948100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921704+0000","last_change":"2024-04-26T22:47:41.784618+0000","last_active":"2024-04-26T22:48:30.921704+0000","last_peered":"2024-04-26T22:48:30.921704+0000","last_clean":"2024-04-26T22:48:30.921704+0000","last_became_active":"2024-04-26T22:47:41.784269+0000","last_became_peered":"2024-04-26T22:47:41.784269+0000","last_unstale":"2024-04-26T22:48:30.921704+0000","last_undegraded":"2024-04-26T22:48:30.921704+0000","last_fullsized":"2024-04-26T22:48:30.921704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:26:08.725744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927636+0000","last_change":"2024-04-26T22:47:41.786063+0000","last_active":"2024-04-26T22:48:30.927636+0000","last_peered":"2024-04-26T22:48:30.927636+0000","last_clean":"2024-04-26T22:48:30.927636+0000","last_became_active":"2024-04-26T22:47:41.785657+0000","last_became_peered":"2024-04-26T22:47:41.785657+0000","last_unstale":"2024-04-26T22:48:30.927636+0000","last_undegraded":"2024-04-26T22:48:30.927636+0000","last_fullsized":"2024-04-26T22:48:30.927636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:14.289477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927261+0000","last_change":"2024-04-26T22:47:41.788788+0000","last_active":"2024-04-26T22:48:30.927261+0000","last_peered":"2024-04-26T22:48:30.927261+0000","last_clean":"2024-04-26T22:48:30.927261+0000","last_became_active":"2024-04-26T22:47:41.788434+0000","last_became_peered":"2024-04-26T22:47:41.788434+0000","last_unstale":"2024-04-26T22:48:30.927261+0000","last_undegraded":"2024-04-26T22:48:30.927261+0000","last_fullsized":"2024-04-26T22:48:30.927261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:05.056380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.919779+0000","last_change":"2024-04-26T22:47:41.780435+0000","last_active":"2024-04-26T22:48:30.919779+0000","last_peered":"2024-04-26T22:48:30.919779+0000","last_clean":"2024-04-26T22:48:30.919779+0000","last_became_active":"2024-04-26T22:47:41.780126+0000","last_became_peered":"2024-04-26T22:47:41.780126+0000","last_unstale":"2024-04-26T22:48:30.919779+0000","last_undegraded":"2024-04-26T22:48:30.919779+0000","last_fullsized":"2024-04-26T22:48:30.919779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:20.061814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993984+0000","last_change":"2024-04-26T22:47:41.775797+0000","last_active":"2024-04-26T22:48:30.993984+0000","last_peered":"2024-04-26T22:48:30.993984+0000","last_clean":"2024-04-26T22:48:30.993984+0000","last_became_active":"2024-04-26T22:47:41.775286+0000","last_became_peered":"2024-04-26T22:47:41.775286+0000","last_unstale":"2024-04-26T22:48:30.993984+0000","last_undegraded":"2024-04-26T22:48:30.993984+0000","last_fullsized":"2024-04-26T22:48:30.993984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:45.290373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921900+0000","last_change":"2024-04-26T22:47:41.784714+0000","last_active":"2024-04-26T22:48:30.921900+0000","last_peered":"2024-04-26T22:48:30.921900+0000","last_clean":"2024-04-26T22:48:30.921900+0000","last_became_active":"2024-04-26T22:47:41.783233+0000","last_became_peered":"2024-04-26T22:47:41.783233+0000","last_unstale":"2024-04-26T22:48:30.921900+0000","last_undegraded":"2024-04-26T22:48:30.921900+0000","last_fullsized":"2024-04-26T22:48:30.921900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:34.350512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.927660+0000","last_change":"2024-04-26T22:47:41.796781+0000","last_active":"2024-04-26T22:48:30.927660+0000","last_peered":"2024-04-26T22:48:30.927660+0000","last_clean":"2024-04-26T22:48:30.927660+0000","last_became_active":"2024-04-26T22:47:41.796326+0000","last_became_peered":"2024-04-26T22:47:41.796326+0000","last_unstale":"2024-04-26T22:48:30.927660+0000","last_undegraded":"2024-04-26T22:48:30.927660+0000","last_fullsized":"2024-04-26T22:48:30.927660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:30:59.924808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993232+0000","last_change":"2024-04-26T22:47:41.785149+0000","last_active":"2024-04-26T22:48:30.993232+0000","last_peered":"2024-04-26T22:48:30.993232+0000","last_clean":"2024-04-26T22:48:30.993232+0000","last_became_active":"2024-04-26T22:47:41.784807+0000","last_became_peered":"2024-04-26T22:47:41.784807+0000","last_unstale":"2024-04-26T22:48:30.993232+0000","last_undegraded":"2024-04-26T22:48:30.993232+0000","last_fullsized":"2024-04-26T22:48:30.993232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:45.521731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923924+0000","last_change":"2024-04-26T22:47:41.768789+0000","last_active":"2024-04-26T22:48:30.923924+0000","last_peered":"2024-04-26T22:48:30.923924+0000","last_clean":"2024-04-26T22:48:30.923924+0000","last_became_active":"2024-04-26T22:47:41.768466+0000","last_became_peered":"2024-04-26T22:47:41.768466+0000","last_unstale":"2024-04-26T22:48:30.923924+0000","last_undegraded":"2024-04-26T22:48:30.923924+0000","last_fullsized":"2024-04-26T22:48:30.923924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:15.798301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.992885+0000","last_change":"2024-04-26T22:47:41.781867+0000","last_active":"2024-04-26T22:48:30.992885+0000","last_peered":"2024-04-26T22:48:30.992885+0000","last_clean":"2024-04-26T22:48:30.992885+0000","last_became_active":"2024-04-26T22:47:41.780648+0000","last_became_peered":"2024-04-26T22:47:41.780648+0000","last_unstale":"2024-04-26T22:48:30.992885+0000","last_undegraded":"2024-04-26T22:48:30.992885+0000","last_fullsized":"2024-04-26T22:48:30.992885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:25.393892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.923684+0000","last_change":"2024-04-26T22:47:41.774622+0000","last_active":"2024-04-26T22:48:30.923684+0000","last_peered":"2024-04-26T22:48:30.923684+0000","last_clean":"2024-04-26T22:48:30.923684+0000","last_became_active":"2024-04-26T22:47:41.774262+0000","last_became_peered":"2024-04-26T22:47:41.774262+0000","last_unstale":"2024-04-26T22:48:30.923684+0000","last_undegraded":"2024-04-26T22:48:30.923684+0000","last_fullsized":"2024-04-26T22:48:30.923684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:32.571497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.993773+0000","last_change":"2024-04-26T22:47:41.768981+0000","last_active":"2024-04-26T22:48:30.993773+0000","last_peered":"2024-04-26T22:48:30.993773+0000","last_clean":"2024-04-26T22:48:30.993773+0000","last_became_active":"2024-04-26T22:47:41.768515+0000","last_became_peered":"2024-04-26T22:47:41.768515+0000","last_unstale":"2024-04-26T22:48:30.993773+0000","last_undegraded":"2024-04-26T22:48:30.993773+0000","last_fullsized":"2024-04-26T22:48:30.993773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:43:47.680248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T22:48:30.921751+0000","last_change":"2024-04-26T22:47:41.788415+0000","last_active":"2024-04-26T22:48:30.921751+0000","last_peered":"2024-04-26T22:48:30.921751+0000","last_clean":"2024-04-26T22:48:30.921751+0000","last_became_active":"2024-04-26T22:47:41.788076+0000","last_became_peered":"2024-04-26T22:47:41.788076+0000","last_unstale":"2024-04-26T22:48:30.921751+0000","last_undegraded":"2024-04-26T22:48:30.921751+0000","last_fullsized":"2024-04-26T22:48:30.921751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T22:47:40.750450+0000","last_clean_scrub_stamp":"2024-04-26T22:47:40.750450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:51:54.658275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":288005,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57602,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":774,"num_read_kb":454,"num_write":232,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":172806,"data_compressed":5538,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":457141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645348,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":1098546,"data_compressed":38462,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":456658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":457014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1049554,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":456375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":507686,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":458105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":508297,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":456200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920834,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":456627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":1048841,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34975,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36616,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":52150,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":52761,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":52525,"data_compressed":1846,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T22:49:04.149 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T22:49:04.150 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T22:49:04.150 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T22:49:04.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T22:49:04.397 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:04 smithi186 ceph-mon[32774]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:49:04.752 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:04 smithi001 ceph-mon[29435]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:49:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:04 smithi038 ceph-mon[32433]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T22:49:05.516 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:05 smithi001 ceph-mon[29435]: pgmap v334: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:05.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:05 smithi186 ceph-mon[32774]: pgmap v334: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:05 smithi038 ceph-mon[32433]: pgmap v334: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 -- 172.21.15.1:0/2142383279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740108380 msgr2=0x7f2740108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 --2- 172.21.15.1:0/2142383279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740108380 0x7f2740108760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2730009990 tx=0x7f2730009ca0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 -- 172.21.15.1:0/2142383279 shutdown_connections 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 --2- 172.21.15.1:0/2142383279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2740102cd0 0x7f274010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 --2- 172.21.15.1:0/2142383279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740108380 0x7f2740108760 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 --2- 172.21.15.1:0/2142383279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2740102310 0x7f2740102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 -- 172.21.15.1:0/2142383279 >> 172.21.15.1:0/2142383279 conn(0x7f2740079290 msgr2=0x7f27400796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 -- 172.21.15.1:0/2142383279 shutdown_connections 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.086+0000 7f2745c0c700 1 -- 172.21.15.1:0/2142383279 wait complete. 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 Processor -- start 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 -- start start 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 0x7f274019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2740102cd0 0x7f274019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2740108380 0x7f27401a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f27401124a0 con 0x7f2740102310 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2740112320 con 0x7f2740108380 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f2745c0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2740112620 con 0x7f2740102cd0 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f273f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 0x7f274019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f273f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 0x7f274019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46756/0 (socket says 172.21.15.1:46756) 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.087+0000 7f273f7fe700 1 -- 172.21.15.1:0/607385944 learned_addr learned my addr 172.21.15.1:0/607385944 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273ffff700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2740108380 0x7f27401a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f2736ffd700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2740102cd0 0x7f274019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273f7fe700 1 -- 172.21.15.1:0/607385944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2740102cd0 msgr2=0x7f274019aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273f7fe700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2740102cd0 0x7f274019aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273f7fe700 1 -- 172.21.15.1:0/607385944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2740108380 msgr2=0x7f27401a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273f7fe700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2740108380 0x7f27401a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273f7fe700 1 -- 172.21.15.1:0/607385944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27401a5f20 con 0x7f2740102310 2024-04-26T22:49:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273f7fe700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 0x7f274019a960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2730000c00 tx=0x7f27300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730027070 con 0x7f2740102310 2024-04-26T22:49:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2730003f20 con 0x7f2740102310 2024-04-26T22:49:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27401a60f0 con 0x7f2740102310 2024-04-26T22:49:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730022cf0 con 0x7f2740102310 2024-04-26T22:49:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.088+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27401a6540 con 0x7f2740102310 2024-04-26T22:49:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.089+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f274006a110 con 0x7f2740102310 2024-04-26T22:49:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.090+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f273002a070 con 0x7f2740102310 2024-04-26T22:49:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.091+0000 7f273d7fa700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f272c0a3cd0 0x7f272c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.091+0000 7f2736ffd700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f272c0a3cd0 0x7f272c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.091+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f27300d12f0 con 0x7f2740102310 2024-04-26T22:49:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.092+0000 7f2736ffd700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f272c0a3cd0 0x7f272c0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2728005950 tx=0x7f27280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.096+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2730013070 con 0x7f2740102310 2024-04-26T22:49:06.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.356+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f274002e650 con 0x7f2740102310 2024-04-26T22:49:06.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.356+0000 7f273d7fa700 1 -- 172.21.15.1:0/607385944 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f273009dc30 con 0x7f2740102310 2024-04-26T22:49:06.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:06.357 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f272c0a3cd0 msgr2=0x7f272c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f272c0a3cd0 0x7f272c0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2728005950 tx=0x7f27280058e0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 msgr2=0x7f274019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 0x7f274019a960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2730000c00 tx=0x7f27300047b0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 shutdown_connections 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f272c0a3cd0 0x7f272c0a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2740108380 0x7f27401a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740102310 0x7f274019a960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 --2- 172.21.15.1:0/607385944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2740102cd0 0x7f274019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 >> 172.21.15.1:0/607385944 conn(0x7f2740079290 msgr2=0x7f27401011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 shutdown_connections 2024-04-26T22:49:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:06.359+0000 7f2745c0c700 1 -- 172.21.15.1:0/607385944 wait complete. 2024-04-26T22:49:06.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:06 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/607385944' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T22:49:06.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:06 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/607385944' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T22:49:06.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:06 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/607385944' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T22:49:07.002 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T22:49:07.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.109+0000 7f2074ac8640 1 -- 172.21.15.1:0/2198687810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20701034d0 msgr2=0x7f20701058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.109+0000 7f2074ac8640 1 --2- 172.21.15.1:0/2198687810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20701034d0 0x7f20701058c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2060009900 tx=0x7f206002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.109+0000 7f2074ac8640 1 -- 172.21.15.1:0/2198687810 shutdown_connections 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 --2- 172.21.15.1:0/2198687810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2070105e00 0x7f20701081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 --2- 172.21.15.1:0/2198687810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2070108730 0x7f207010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 --2- 172.21.15.1:0/2198687810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20701034d0 0x7f20701058c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 -- 172.21.15.1:0/2198687810 >> 172.21.15.1:0/2198687810 conn(0x7f20700fce00 msgr2=0x7f20700ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 -- 172.21.15.1:0/2198687810 shutdown_connections 2024-04-26T22:49:07.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 -- 172.21.15.1:0/2198687810 wait complete. 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 Processor -- start 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 -- start start 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20701034d0 0x7f2070102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2070105e00 0x7f2070101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.110+0000 7f2074ac8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 0x7f20701019c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f2074ac8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f207010e4c0 con 0x7f2070108730 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f2074ac8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f207010e340 con 0x7f20701034d0 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f2074ac8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f207010e640 con 0x7f2070105e00 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 0x7f20701019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 0x7f20701019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46776/0 (socket says 172.21.15.1:46776) 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 -- 172.21.15.1:0/1079899718 learned_addr learned my addr 172.21.15.1:0/1079899718 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206dd9b640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2070105e00 0x7f2070101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206e59c640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20701034d0 0x7f2070102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 -- 172.21.15.1:0/1079899718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2070105e00 msgr2=0x7f2070101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2070105e00 0x7f2070101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 -- 172.21.15.1:0/1079899718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20701034d0 msgr2=0x7f2070102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20701034d0 0x7f2070102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 -- 172.21.15.1:0/1079899718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2070102250 con 0x7f2070108730 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f206ed9d640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 0x7f20701019c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2068009fd0 tx=0x7f206800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068017070 con 0x7f2070108730 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20701a2d10 con 0x7f2070108730 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2068003e20 con 0x7f2070108730 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.111+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20680106c0 con 0x7f2070108730 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.112+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20701a31b0 con 0x7f2070108730 2024-04-26T22:49:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.113+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20700fe8f0 con 0x7f2070108730 2024-04-26T22:49:07.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.114+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2068004140 con 0x7f2070108730 2024-04-26T22:49:07.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.115+0000 7f20677fe640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f205c0a3c40 0x7f205c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.115+0000 7f206e59c640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f205c0a3c40 0x7f205c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.115+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f20680d0180 con 0x7f2070108730 2024-04-26T22:49:07.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.115+0000 7f206e59c640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f205c0a3c40 0x7f205c0a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2060000c00 tx=0x7f20600314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.119+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f20680047e0 con 0x7f2070108730 2024-04-26T22:49:07.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.308+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f207002dd20 con 0x7f2070108730 2024-04-26T22:49:07.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.309+0000 7f20677fe640 1 -- 172.21.15.1:0/1079899718 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f206809c3e0 con 0x7f2070108730 2024-04-26T22:49:07.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:07.310 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:49:07.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f205c0a3c40 msgr2=0x7f205c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f205c0a3c40 0x7f205c0a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2060000c00 tx=0x7f20600314a0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 msgr2=0x7f20701019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 0x7f20701019c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2068009fd0 tx=0x7f206800b260 comp rx=0 tx=0).stop 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 shutdown_connections 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f205c0a3c40 0x7f205c0a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2070105e00 0x7f2070101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2070108730 0x7f20701019c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 --2- 172.21.15.1:0/1079899718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20701034d0 0x7f2070102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 >> 172.21.15.1:0/1079899718 conn(0x7f20700fce00 msgr2=0x7f2070103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 shutdown_connections 2024-04-26T22:49:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.312+0000 7f2074ac8640 1 -- 172.21.15.1:0/1079899718 wait complete. 2024-04-26T22:49:07.324 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T22:47:39.631594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T22:47:40.081969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:49:07.325 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 -- 172.21.15.1:0/1130938001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 msgr2=0x7f1d68102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 --2- 172.21.15.1:0/1130938001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68102db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009ca0 tx=0x7f1d5802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 -- 172.21.15.1:0/1130938001 shutdown_connections 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 --2- 172.21.15.1:0/1130938001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d68104540 0x7f1d6810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 --2- 172.21.15.1:0/1130938001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d68103b80 0x7f1d68104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 --2- 172.21.15.1:0/1130938001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68102db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 -- 172.21.15.1:0/1130938001 >> 172.21.15.1:0/1130938001 conn(0x7f1d680fdf00 msgr2=0x7f1d68100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.429+0000 7f1d6e247640 1 -- 172.21.15.1:0/1130938001 shutdown_connections 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 -- 172.21.15.1:0/1130938001 wait complete. 2024-04-26T22:49:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 Processor -- start 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 -- start start 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d68103b80 0x7f1d68199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d68104540 0x7f1d681a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6810e490 con 0x7f1d68102990 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6810e310 con 0x7f1d68104540 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d6e247640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6810e610 con 0x7f1d68103b80 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46800/0 (socket says 172.21.15.1:46800) 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.430+0000 7f1d677fe640 1 -- 172.21.15.1:0/776091669 learned_addr learned my addr 172.21.15.1:0/776091669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d67fff640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d68104540 0x7f1d681a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d677fe640 1 -- 172.21.15.1:0/776091669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d68103b80 msgr2=0x7f1d68199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d66ffd640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d68103b80 0x7f1d68199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d677fe640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d68103b80 0x7f1d68199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d677fe640 1 -- 172.21.15.1:0/776091669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d68104540 msgr2=0x7f1d681a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d677fe640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d68104540 0x7f1d681a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d677fe640 1 -- 172.21.15.1:0/776091669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d681a07e0 con 0x7f1d68102990 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d677fe640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68198bc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009430 tx=0x7f1d58031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58003cb0 con 0x7f1d68102990 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d58003e50 con 0x7f1d68102990 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d681a0a70 con 0x7f1d68102990 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5802fc60 con 0x7f1d68102990 2024-04-26T22:49:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.431+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d681a0fc0 con 0x7f1d68102990 2024-04-26T22:49:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.432+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d68109d80 con 0x7f1d68102990 2024-04-26T22:49:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.433+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1d58004040 con 0x7f1d68102990 2024-04-26T22:49:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.434+0000 7f1d64ff9640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1d540a3c40 0x7f1d540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.434+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f1d580f1c90 con 0x7f1d68102990 2024-04-26T22:49:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.434+0000 7f1d66ffd640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1d540a3c40 0x7f1d540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.434+0000 7f1d66ffd640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1d540a3c40 0x7f1d540a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1d50005fd0 tx=0x7f1d50005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.437+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d580be5f0 con 0x7f1d68102990 2024-04-26T22:49:07.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:07 smithi001 ceph-mon[29435]: pgmap v335: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T22:49:07.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:07 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1079899718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.624+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d6802dd20 con 0x7f1d68102990 2024-04-26T22:49:07.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.625+0000 7f1d64ff9640 1 -- 172.21.15.1:0/776091669 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f1d580bdef0 con 0x7f1d68102990 2024-04-26T22:49:07.625 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1d540a3c40 msgr2=0x7f1d540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1d540a3c40 0x7f1d540a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1d50005fd0 tx=0x7f1d50005850 comp rx=0 tx=0).stop 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 msgr2=0x7f1d68198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68198bc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009430 tx=0x7f1d58031840 comp rx=0 tx=0).stop 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 shutdown_connections 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1d540a3c40 0x7f1d540a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d68103b80 0x7f1d68199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d68104540 0x7f1d681a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 --2- 172.21.15.1:0/776091669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d68102990 0x7f1d68198bc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 >> 172.21.15.1:0/776091669 conn(0x7f1d680fdf00 msgr2=0x7f1d68106f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:07.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 shutdown_connections 2024-04-26T22:49:07.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.627+0000 7f1d6e247640 1 -- 172.21.15.1:0/776091669 wait complete. 2024-04-26T22:49:07.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T22:49:07.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:07 smithi186 ceph-mon[32774]: pgmap v335: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T22:49:07.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:07 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1079899718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:07.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 -- 172.21.15.1:0/1757356600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e01086d0 msgr2=0x7f86e010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/1757356600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e01086d0 0x7f86e010aac0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009ca0 tx=0x7f86d802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 -- 172.21.15.1:0/1757356600 shutdown_connections 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/1757356600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e01086d0 0x7f86e010aac0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/1757356600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86e0105da0 0x7f86e0108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/1757356600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e0103470 0x7f86e0105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 -- 172.21.15.1:0/1757356600 >> 172.21.15.1:0/1757356600 conn(0x7f86e00fce00 msgr2=0x7f86e00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.743+0000 7f86e6ee1640 1 -- 172.21.15.1:0/1757356600 shutdown_connections 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 -- 172.21.15.1:0/1757356600 wait complete. 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 Processor -- start 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 -- start start 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86e0103470 0x7f86e0198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 0x7f86e01990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e01086d0 0x7f86e01a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f86e010e4a0 con 0x7f86e0105da0 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f86e010e320 con 0x7f86e0103470 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e6ee1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86e010e620 con 0x7f86e01086d0 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 0x7f86e01990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 0x7f86e01990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46824/0 (socket says 172.21.15.1:46824) 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86dffff640 1 -- 172.21.15.1:0/903953607 learned_addr learned my addr 172.21.15.1:0/903953607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.744+0000 7f86e4c7d640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86e0103470 0x7f86e0198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86e547e640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e01086d0 0x7f86e01a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86dffff640 1 -- 172.21.15.1:0/903953607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e01086d0 msgr2=0x7f86e01a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86dffff640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e01086d0 0x7f86e01a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86dffff640 1 -- 172.21.15.1:0/903953607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86e0103470 msgr2=0x7f86e0198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86dffff640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86e0103470 0x7f86e0198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86dffff640 1 -- 172.21.15.1:0/903953607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e01a07c0 con 0x7f86e0105da0 2024-04-26T22:49:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86dffff640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 0x7f86e01990e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f86d400d770 tx=0x7f86d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4017070 con 0x7f86e0105da0 2024-04-26T22:49:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d4009c30 con 0x7f86e0105da0 2024-04-26T22:49:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e01a0ab0 con 0x7f86e0105da0 2024-04-26T22:49:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d400f620 con 0x7f86e0105da0 2024-04-26T22:49:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.745+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e01a1000 con 0x7f86e0105da0 2024-04-26T22:49:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.746+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e00fe8f0 con 0x7f86e0105da0 2024-04-26T22:49:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.747+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f86d401a070 con 0x7f86e0105da0 2024-04-26T22:49:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.748+0000 7f86ddffb640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f86c80a3c40 0x7f86c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.748+0000 7f86e4c7d640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f86c80a3c40 0x7f86c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.748+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f86d40d0100 con 0x7f86e0105da0 2024-04-26T22:49:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.749+0000 7f86e4c7d640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f86c80a3c40 0x7f86c80a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f86d00058c0 tx=0x7f86d000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.753+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f86d4009de0 con 0x7f86e0105da0 2024-04-26T22:49:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:07 smithi038 ceph-mon[32433]: pgmap v335: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T22:49:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:07 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1079899718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:07.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.944+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86e002dd10 con 0x7f86e0105da0 2024-04-26T22:49:07.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.944+0000 7f86ddffb640 1 -- 172.21.15.1:0/903953607 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f86d409c950 con 0x7f86e0105da0 2024-04-26T22:49:07.944 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f86c80a3c40 msgr2=0x7f86c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f86c80a3c40 0x7f86c80a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f86d00058c0 tx=0x7f86d000a2d0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 msgr2=0x7f86e01990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 0x7f86e01990e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f86d400d770 tx=0x7f86d400dc30 comp rx=0 tx=0).stop 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 shutdown_connections 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f86c80a3c40 0x7f86c80a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e01086d0 0x7f86e01a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86e0105da0 0x7f86e01990e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 --2- 172.21.15.1:0/903953607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86e0103470 0x7f86e0198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.946+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 >> 172.21.15.1:0/903953607 conn(0x7f86e00fce00 msgr2=0x7f86e0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.947+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 shutdown_connections 2024-04-26T22:49:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:07.947+0000 7f86e6ee1640 1 -- 172.21.15.1:0/903953607 wait complete. 2024-04-26T22:49:07.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T22:49:08.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 -- 172.21.15.1:0/3720458977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218103410 msgr2=0x7f9218105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 --2- 172.21.15.1:0/3720458977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218103410 0x7f9218105800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9208009900 tx=0x7f920802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 -- 172.21.15.1:0/3720458977 shutdown_connections 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 --2- 172.21.15.1:0/3720458977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9218105d40 0x7f9218108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 --2- 172.21.15.1:0/3720458977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9218108670 0x7f921810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 --2- 172.21.15.1:0/3720458977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218103410 0x7f9218105800 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 -- 172.21.15.1:0/3720458977 >> 172.21.15.1:0/3720458977 conn(0x7f92180fce00 msgr2=0x7f92180ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 -- 172.21.15.1:0/3720458977 shutdown_connections 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.063+0000 7f921da85640 1 -- 172.21.15.1:0/3720458977 wait complete. 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 Processor -- start 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 -- start start 2024-04-26T22:49:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9218103410 0x7f9218198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 0x7f9218199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9218108670 0x7f92181a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f921810e4c0 con 0x7f9218105d40 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f921810e340 con 0x7f9218108670 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f921da85640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f921810e640 con 0x7f9218103410 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 0x7f9218199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 0x7f9218199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46838/0 (socket says 172.21.15.1:46838) 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 -- 172.21.15.1:0/1112993688 learned_addr learned my addr 172.21.15.1:0/1112993688 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9217fff640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9218108670 0x7f92181a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 -- 172.21.15.1:0/1112993688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9218103410 msgr2=0x7f9218198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9218103410 0x7f9218198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 -- 172.21.15.1:0/1112993688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9218108670 msgr2=0x7f92181a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9218108670 0x7f92181a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.064+0000 7f9216ffd640 1 -- 172.21.15.1:0/1112993688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92181a0830 con 0x7f9218105d40 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.065+0000 7f9216ffd640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 0x7f9218199160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f920c00d770 tx=0x7f920c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.065+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c017070 con 0x7f9218105d40 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.065+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92181a0b20 con 0x7f9218105d40 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.065+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f920c009c30 con 0x7f9218105d40 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.065+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c00f620 con 0x7f9218105d40 2024-04-26T22:49:08.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.065+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92181a10c0 con 0x7f9218105d40 2024-04-26T22:49:08.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.066+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92180fe8f0 con 0x7f9218105d40 2024-04-26T22:49:08.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.067+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f920c01a070 con 0x7f9218105d40 2024-04-26T22:49:08.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.067+0000 7f9214ff9640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f92000a3c40 0x7f92000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.068+0000 7f92177fe640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f92000a3c40 0x7f92000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.068+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f920c0d0240 con 0x7f9218105d40 2024-04-26T22:49:08.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.068+0000 7f92177fe640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f92000a3c40 0x7f92000a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92080096c0 tx=0x7f9208000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:08.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.072+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f920c009de0 con 0x7f9218105d40 2024-04-26T22:49:08.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.260+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f921802dca0 con 0x7f9218105d40 2024-04-26T22:49:08.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.260+0000 7f9214ff9640 1 -- 172.21.15.1:0/1112993688 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f920c09ca90 con 0x7f9218105d40 2024-04-26T22:49:08.260 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f92000a3c40 msgr2=0x7f92000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f92000a3c40 0x7f92000a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92080096c0 tx=0x7f9208000bc0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 msgr2=0x7f9218199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 0x7f9218199160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f920c00d770 tx=0x7f920c00dc30 comp rx=0 tx=0).stop 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 shutdown_connections 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f92000a3c40 0x7f92000a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9218108670 0x7f92181a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9218105d40 0x7f9218199160 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 --2- 172.21.15.1:0/1112993688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9218103410 0x7f9218198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.262+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 >> 172.21.15.1:0/1112993688 conn(0x7f92180fce00 msgr2=0x7f9218105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.263+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 shutdown_connections 2024-04-26T22:49:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.263+0000 7f921da85640 1 -- 172.21.15.1:0/1112993688 wait complete. 2024-04-26T22:49:08.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:49:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.375+0000 7fc0c24fc640 1 -- 172.21.15.1:0/1393403751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 msgr2=0x7fc0bc105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.375+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/1393403751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc105800 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009900 tx=0x7fc0ac02a410 comp rx=0 tx=0).stop 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- 172.21.15.1:0/1393403751 shutdown_connections 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/1393403751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0bc105d40 0x7fc0bc108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/1393403751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0bc108670 0x7fc0bc10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/1393403751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc105800 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- 172.21.15.1:0/1393403751 >> 172.21.15.1:0/1393403751 conn(0x7fc0bc0fce40 msgr2=0x7fc0bc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- 172.21.15.1:0/1393403751 shutdown_connections 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- 172.21.15.1:0/1393403751 wait complete. 2024-04-26T22:49:08.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 Processor -- start 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- start start 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0bc105d40 0x7fc0bc1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0bc108670 0x7fc0bc101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc10e590 con 0x7fc0bc103410 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc10e410 con 0x7fc0bc105d40 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.376+0000 7fc0c24fc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc10e710 con 0x7fc0bc108670 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46854/0 (socket says 172.21.15.1:46854) 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 -- 172.21.15.1:0/4102690868 learned_addr learned my addr 172.21.15.1:0/4102690868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bb7fe640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0bc105d40 0x7fc0bc1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0c0a99640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0bc108670 0x7fc0bc101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 -- 172.21.15.1:0/4102690868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0bc108670 msgr2=0x7fc0bc101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0bc108670 0x7fc0bc101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 -- 172.21.15.1:0/4102690868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0bc105d40 msgr2=0x7fc0bc1013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0bc105d40 0x7fc0bc1013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 -- 172.21.15.1:0/4102690868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc102190 con 0x7fc0bc103410 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0bbfff640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc102d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac005ab0 tx=0x7fc0ac003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac03f070 con 0x7fc0bc103410 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0ac03b460 con 0x7fc0bc103410 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0bc1a2d20 con 0x7fc0bc103410 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac035030 con 0x7fc0bc103410 2024-04-26T22:49:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.377+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0bc1a3230 con 0x7fc0bc103410 2024-04-26T22:49:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.378+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0bc0fe930 con 0x7fc0bc103410 2024-04-26T22:49:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.379+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc0ac042070 con 0x7fc0bc103410 2024-04-26T22:49:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.380+0000 7fc0b97fa640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a40a3cc0 0x7fc0a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.380+0000 7fc0bb7fe640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a40a3cc0 0x7fc0a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.380+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc0ac0f11c0 con 0x7fc0bc103410 2024-04-26T22:49:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.381+0000 7fc0bb7fe640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a40a3cc0 0x7fc0a40a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc102af0 tx=0x7fc0b0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:08.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.383+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0ac0bda70 con 0x7fc0bc103410 2024-04-26T22:49:08.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:08 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/776091669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:08 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/903953607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:08 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1112993688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.609+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0bc02dad0 con 0x7fc0bc103410 2024-04-26T22:49:08.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.610+0000 7fc0b97fa640 1 -- 172.21.15.1:0/4102690868 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11917 (secure 0 0 0) 0x7fc0ac0bd370 con 0x7fc0bc103410 2024-04-26T22:49:08.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:08.610 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:47:47.801060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[24475,24514,24487,24473,14631]},"id":1}]} 2024-04-26T22:49:08.610 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a40a3cc0 msgr2=0x7fc0a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a40a3cc0 0x7fc0a40a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc102af0 tx=0x7fc0b0009110 comp rx=0 tx=0).stop 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 msgr2=0x7fc0bc102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc102d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac005ab0 tx=0x7fc0ac003cd0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 shutdown_connections 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc0a40a3cc0 0x7fc0a40a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0bc108670 0x7fc0bc101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0bc105d40 0x7fc0bc1013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 --2- 172.21.15.1:0/4102690868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc103410 0x7fc0bc102d80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 >> 172.21.15.1:0/4102690868 conn(0x7fc0bc0fce40 msgr2=0x7fc0bc103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:08.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 shutdown_connections 2024-04-26T22:49:08.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.612+0000 7fc0c24fc640 1 -- 172.21.15.1:0/4102690868 wait complete. 2024-04-26T22:49:08.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.724+0000 7f6e12d59640 1 -- 172.21.15.1:0/2381402512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c105d40 msgr2=0x7f6e0c108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.724+0000 7f6e12d59640 1 --2- 172.21.15.1:0/2381402512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c105d40 0x7f6e0c108130 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009900 tx=0x7f6e0002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- 172.21.15.1:0/2381402512 shutdown_connections 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 --2- 172.21.15.1:0/2381402512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c108670 0x7f6e0c10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 --2- 172.21.15.1:0/2381402512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c105d40 0x7f6e0c108130 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 --2- 172.21.15.1:0/2381402512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e0c103410 0x7f6e0c105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- 172.21.15.1:0/2381402512 >> 172.21.15.1:0/2381402512 conn(0x7f6e0c0fce00 msgr2=0x7f6e0c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- 172.21.15.1:0/2381402512 shutdown_connections 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- 172.21.15.1:0/2381402512 wait complete. 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 Processor -- start 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- start start 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c103410 0x7f6e0c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e0c105d40 0x7f6e0c199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 0x7f6e0c1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c10e530 con 0x7f6e0c108670 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c10e3b0 con 0x7f6e0c103410 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e12d59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c10e6b0 con 0x7f6e0c105d40 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.725+0000 7f6e112f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 0x7f6e0c1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 0x7f6e0c1a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46868/0 (socket says 172.21.15.1:46868) 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 -- 172.21.15.1:0/3271871620 learned_addr learned my addr 172.21.15.1:0/3271871620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e10af5640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c103410 0x7f6e0c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e0bfff640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e0c105d40 0x7f6e0c199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 -- 172.21.15.1:0/3271871620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e0c105d40 msgr2=0x7f6e0c199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e0c105d40 0x7f6e0c199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 -- 172.21.15.1:0/3271871620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c103410 msgr2=0x7f6e0c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c103410 0x7f6e0c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 -- 172.21.15.1:0/3271871620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0c1a0860 con 0x7f6e0c108670 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e112f6640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 0x7f6e0c1a01c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400b970 tx=0x7f6e0400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e04019070 con 0x7f6e0c108670 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e04014440 con 0x7f6e0c108670 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0c1a0b50 con 0x7f6e0c108670 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e040075d0 con 0x7f6e0c108670 2024-04-26T22:49:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.726+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e0c1a1050 con 0x7f6e0c108670 2024-04-26T22:49:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.727+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e0c0fe8f0 con 0x7f6e0c108670 2024-04-26T22:49:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.728+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f6e0401c070 con 0x7f6e0c108670 2024-04-26T22:49:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.729+0000 7f6e09ffb640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6df40a3c40 0x7f6df40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.729+0000 7f6e10af5640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6df40a3c40 0x7f6df40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.729+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f6e040cfa60 con 0x7f6e0c108670 2024-04-26T22:49:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.729+0000 7f6e10af5640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6df40a3c40 0x7f6df40a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0058c0 tx=0x7f6dfc00a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:08.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.733+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6e0409bdd0 con 0x7f6e0c108670 2024-04-26T22:49:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:08 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/776091669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:08 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/903953607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:08 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1112993688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:08 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/776091669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:08 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/903953607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:08 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1112993688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:08.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.922+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e0c02dca0 con 0x7f6e0c108670 2024-04-26T22:49:08.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.924+0000 7f6e09ffb640 1 -- 172.21.15.1:0/3271871620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f6e040a00d0 con 0x7f6e0c108670 2024-04-26T22:49:08.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:49:08.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6df40a3c40 msgr2=0x7f6df40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6df40a3c40 0x7f6df40a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0058c0 tx=0x7f6dfc00a250 comp rx=0 tx=0).stop 2024-04-26T22:49:08.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 msgr2=0x7f6e0c1a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:08.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 0x7f6e0c1a01c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400b970 tx=0x7f6e0400be30 comp rx=0 tx=0).stop 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 shutdown_connections 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6df40a3c40 0x7f6df40a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e0c105d40 0x7f6e0c199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e0c108670 0x7f6e0c1a01c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 --2- 172.21.15.1:0/3271871620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c103410 0x7f6e0c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 >> 172.21.15.1:0/3271871620 conn(0x7f6e0c0fce00 msgr2=0x7f6e0c103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 shutdown_connections 2024-04-26T22:49:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:08.926+0000 7f6e12d59640 1 -- 172.21.15.1:0/3271871620 wait complete. 2024-04-26T22:49:08.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T22:47:39.631594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T22:47:40.081969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:49:08.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T22:49:09.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 -- 172.21.15.1:0/3784897330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544102a70 msgr2=0x7f3544102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 --2- 172.21.15.1:0/3784897330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544102a70 0x7f3544102ef0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3538009900 tx=0x7f353802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 -- 172.21.15.1:0/3784897330 shutdown_connections 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 --2- 172.21.15.1:0/3784897330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3544103430 0x7f3544109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 --2- 172.21.15.1:0/3784897330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544102a70 0x7f3544102ef0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 --2- 172.21.15.1:0/3784897330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 0x7f3544101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 -- 172.21.15.1:0/3784897330 >> 172.21.15.1:0/3784897330 conn(0x7f35440fce10 msgr2=0x7f35440ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 -- 172.21.15.1:0/3784897330 shutdown_connections 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 -- 172.21.15.1:0/3784897330 wait complete. 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 Processor -- start 2024-04-26T22:49:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.043+0000 7f354a339640 1 -- start start 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 0x7f3544198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3544102a70 0x7f3544199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 0x7f35441a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f354410d3b0 con 0x7f3544103430 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f354410d230 con 0x7f3544101880 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f354410d530 con 0x7f3544102a70 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 0x7f35441a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 0x7f35441a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46886/0 (socket says 172.21.15.1:46886) 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 -- 172.21.15.1:0/2428068479 learned_addr learned my addr 172.21.15.1:0/2428068479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35437fe640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3544102a70 0x7f3544199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f3543fff640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 0x7f3544198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 -- 172.21.15.1:0/2428068479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3544102a70 msgr2=0x7f3544199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3544102a70 0x7f3544199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 -- 172.21.15.1:0/2428068479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 msgr2=0x7f3544198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 0x7f3544198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 -- 172.21.15.1:0/2428068479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35441a0800 con 0x7f3544103430 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f3543fff640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 0x7f3544198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35488d6640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 0x7f35441a0190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f353c00eaa0 tx=0x7f353c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c00cdb0 con 0x7f3544103430 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35441a0af0 con 0x7f3544103430 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f353c005020 con 0x7f3544103430 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.044+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0116d0 con 0x7f3544103430 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.045+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441a1010 con 0x7f3544103430 2024-04-26T22:49:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.046+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35441a1240 con 0x7f3544103430 2024-04-26T22:49:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.047+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f353c0054a0 con 0x7f3544103430 2024-04-26T22:49:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.047+0000 7f35417fa640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f352c0a3c40 0x7f352c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.047+0000 7f3543fff640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f352c0a3c40 0x7f352c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.047+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f353c0d0b00 con 0x7f3544103430 2024-04-26T22:49:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.048+0000 7f3543fff640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f352c0a3c40 0x7f352c0a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f35441028d0 tx=0x7f3534005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.051+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f353c016030 con 0x7f3544103430 2024-04-26T22:49:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.237+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f35441a1410 con 0x7f3544103430 2024-04-26T22:49:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.237+0000 7f35417fa640 1 -- 172.21.15.1:0/2428068479 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f353c09d3b0 con 0x7f3544103430 2024-04-26T22:49:09.238 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f352c0a3c40 msgr2=0x7f352c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f352c0a3c40 0x7f352c0a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f35441028d0 tx=0x7f3534005f90 comp rx=0 tx=0).stop 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 msgr2=0x7f35441a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 0x7f35441a0190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f353c00eaa0 tx=0x7f353c00ef60 comp rx=0 tx=0).stop 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 shutdown_connections 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f352c0a3c40 0x7f352c0a6100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3544102a70 0x7f3544199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544103430 0x7f35441a0190 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 --2- 172.21.15.1:0/2428068479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544101880 0x7f3544198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 >> 172.21.15.1:0/2428068479 conn(0x7f35440fce10 msgr2=0x7f35440fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 shutdown_connections 2024-04-26T22:49:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.239+0000 7f354a339640 1 -- 172.21.15.1:0/2428068479 wait complete. 2024-04-26T22:49:09.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T22:49:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 -- 172.21.15.1:0/1493253834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0103400 msgr2=0x7f31a01057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 --2- 172.21.15.1:0/1493253834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0103400 0x7f31a01057f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3190009900 tx=0x7f319002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 -- 172.21.15.1:0/1493253834 shutdown_connections 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 --2- 172.21.15.1:0/1493253834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a0105d30 0x7f31a0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 --2- 172.21.15.1:0/1493253834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a0108660 0x7f31a010aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 --2- 172.21.15.1:0/1493253834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0103400 0x7f31a01057f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 -- 172.21.15.1:0/1493253834 >> 172.21.15.1:0/1493253834 conn(0x7f31a00fce10 msgr2=0x7f31a00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 -- 172.21.15.1:0/1493253834 shutdown_connections 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.355+0000 7f31a6197640 1 -- 172.21.15.1:0/1493253834 wait complete. 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 Processor -- start 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 -- start start 2024-04-26T22:49:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a0103400 0x7f31a0198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a0105d30 0x7f31a0199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 0x7f31a01a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f31a010e4e0 con 0x7f31a0108660 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31a010e360 con 0x7f31a0103400 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f31a6197640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31a010e660 con 0x7f31a0105d30 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.356+0000 7f319ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 0x7f31a01a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 0x7f31a01a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46900/0 (socket says 172.21.15.1:46900) 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 -- 172.21.15.1:0/3918957370 learned_addr learned my addr 172.21.15.1:0/3918957370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319effd640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a0105d30 0x7f31a0199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319f7fe640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a0103400 0x7f31a0198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 -- 172.21.15.1:0/3918957370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a0105d30 msgr2=0x7f31a0199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a0105d30 0x7f31a0199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 -- 172.21.15.1:0/3918957370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a0103400 msgr2=0x7f31a0198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a0103400 0x7f31a0198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 -- 172.21.15.1:0/3918957370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01a0840 con 0x7f31a0108660 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319ffff640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 0x7f31a01a01a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f319800eb10 tx=0x7f319800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198004b20 con 0x7f31a0108660 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a01a0b30 con 0x7f31a0108660 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3198004cc0 con 0x7f31a0108660 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198010670 con 0x7f31a0108660 2024-04-26T22:49:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.357+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01a1080 con 0x7f31a0108660 2024-04-26T22:49:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.358+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a00fe900 con 0x7f31a0108660 2024-04-26T22:49:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.359+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3198005000 con 0x7f31a0108660 2024-04-26T22:49:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.360+0000 7f319cff9640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f31880a3c40 0x7f31880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.360+0000 7f319f7fe640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f31880a3c40 0x7f31880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.360+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f31980d0ed0 con 0x7f31a0108660 2024-04-26T22:49:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.360+0000 7f319f7fe640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f31880a3c40 0x7f31880a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f319002a930 tx=0x7f3190000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.363+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3198054060 con 0x7f31a0108660 2024-04-26T22:49:09.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.549+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f31a002dad0 con 0x7f31a0108660 2024-04-26T22:49:09.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.549+0000 7f319cff9640 1 -- 172.21.15.1:0/3918957370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f319809d780 con 0x7f31a0108660 2024-04-26T22:49:09.550 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f31880a3c40 msgr2=0x7f31880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f31880a3c40 0x7f31880a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f319002a930 tx=0x7f3190000bc0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 msgr2=0x7f31a01a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 0x7f31a01a01a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f319800eb10 tx=0x7f319800c2d0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 shutdown_connections 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f31880a3c40 0x7f31880a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a0105d30 0x7f31a0199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0108660 0x7f31a01a01a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 --2- 172.21.15.1:0/3918957370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a0103400 0x7f31a0198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 >> 172.21.15.1:0/3918957370 conn(0x7f31a00fce10 msgr2=0x7f31a0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 shutdown_connections 2024-04-26T22:49:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.551+0000 7f31a6197640 1 -- 172.21.15.1:0/3918957370 wait complete. 2024-04-26T22:49:09.561 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:09 smithi001 ceph-mon[29435]: pgmap v336: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T22:49:09.562 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:09 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4102690868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:09.562 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:09 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3271871620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:09.562 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:09 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2428068479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:09.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T22:49:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 -- 172.21.15.1:0/3190154476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4104540 msgr2=0x7f0ae410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/3190154476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4104540 0x7f0ae410add0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009ca0 tx=0x7f0adc02a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 -- 172.21.15.1:0/3190154476 shutdown_connections 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/3190154476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4104540 0x7f0ae410add0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/3190154476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ae4103b80 0x7f0ae4104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/3190154476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4102990 0x7f0ae4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 -- 172.21.15.1:0/3190154476 >> 172.21.15.1:0/3190154476 conn(0x7f0ae40fdf00 msgr2=0x7f0ae4100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 -- 172.21.15.1:0/3190154476 shutdown_connections 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.667+0000 7f0ae9d89640 1 -- 172.21.15.1:0/3190154476 wait complete. 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 Processor -- start 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 -- start start 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 0x7f0ae4198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4103b80 0x7f0ae4199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ae4104540 0x7f0ae41a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae410e5b0 con 0x7f0ae4102990 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae410e430 con 0x7f0ae4104540 2024-04-26T22:49:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae9d89640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae410e730 con 0x7f0ae4103b80 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 0x7f0ae4198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 0x7f0ae4198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46920/0 (socket says 172.21.15.1:46920) 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae37fe640 1 -- 172.21.15.1:0/812283228 learned_addr learned my addr 172.21.15.1:0/812283228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.668+0000 7f0ae3fff640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ae4104540 0x7f0ae41a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae37fe640 1 -- 172.21.15.1:0/812283228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4103b80 msgr2=0x7f0ae4199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae2ffd640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4103b80 0x7f0ae4199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae37fe640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4103b80 0x7f0ae4199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae37fe640 1 -- 172.21.15.1:0/812283228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ae4104540 msgr2=0x7f0ae41a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae37fe640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ae4104540 0x7f0ae41a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae37fe640 1 -- 172.21.15.1:0/812283228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae41a0960 con 0x7f0ae4102990 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae2ffd640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4103b80 0x7f0ae4199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae37fe640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 0x7f0ae4198d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400b450 tx=0x7f0ad400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4015020 con 0x7f0ae4102990 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ad4011d40 con 0x7f0ae4102990 2024-04-26T22:49:09.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae41a0c50 con 0x7f0ae4102990 2024-04-26T22:49:09.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad400f340 con 0x7f0ae4102990 2024-04-26T22:49:09.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.669+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41a1150 con 0x7f0ae4102990 2024-04-26T22:49:09.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.670+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae4109dd0 con 0x7f0ae4102990 2024-04-26T22:49:09.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.671+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f0ad4018020 con 0x7f0ae4102990 2024-04-26T22:49:09.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.672+0000 7f0ae0ff9640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0acc0a3c40 0x7f0acc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.672+0000 7f0ae2ffd640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0acc0a3c40 0x7f0acc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.672+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f0ad40d0010 con 0x7f0ae4102990 2024-04-26T22:49:09.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.672+0000 7f0ae2ffd640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0acc0a3c40 0x7f0acc0a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8006fd0 tx=0x7f0ad8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.676+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0ad401f020 con 0x7f0ae4102990 2024-04-26T22:49:09.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:09 smithi186 ceph-mon[32774]: pgmap v336: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T22:49:09.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:09 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/4102690868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:09.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:09 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3271871620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:09.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:09 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2428068479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:09 smithi038 ceph-mon[32433]: pgmap v336: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T22:49:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:09 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/4102690868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:09 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3271871620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:09 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2428068479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:09.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.866+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0ae402dc80 con 0x7f0ae4102990 2024-04-26T22:49:09.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.866+0000 7f0ae0ff9640 1 -- 172.21.15.1:0/812283228 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f0ad400d040 con 0x7f0ae4102990 2024-04-26T22:49:09.866 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0acc0a3c40 msgr2=0x7f0acc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0acc0a3c40 0x7f0acc0a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8006fd0 tx=0x7f0ad8008040 comp rx=0 tx=0).stop 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 msgr2=0x7f0ae4198d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 0x7f0ae4198d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400b450 tx=0x7f0ad400b910 comp rx=0 tx=0).stop 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 shutdown_connections 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f0acc0a3c40 0x7f0acc0a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ae4103b80 0x7f0ae4199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ae4104540 0x7f0ae41a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 --2- 172.21.15.1:0/812283228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ae4102990 0x7f0ae4198d40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 >> 172.21.15.1:0/812283228 conn(0x7f0ae40fdf00 msgr2=0x7f0ae4106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 shutdown_connections 2024-04-26T22:49:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.868+0000 7f0ae9d89640 1 -- 172.21.15.1:0/812283228 wait complete. 2024-04-26T22:49:09.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:49:09.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 -- 172.21.15.1:0/403621848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af01045d0 msgr2=0x7f6af010ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 --2- 172.21.15.1:0/403621848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af01045d0 0x7f6af010ae60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009ca0 tx=0x7f6ae802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 -- 172.21.15.1:0/403621848 shutdown_connections 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 --2- 172.21.15.1:0/403621848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af01045d0 0x7f6af010ae60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 --2- 172.21.15.1:0/403621848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af0103c10 0x7f6af0104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 --2- 172.21.15.1:0/403621848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0102a20 0x7f6af0102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 -- 172.21.15.1:0/403621848 >> 172.21.15.1:0/403621848 conn(0x7f6af00fdf90 msgr2=0x7f6af01003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 -- 172.21.15.1:0/403621848 shutdown_connections 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.981+0000 7f6af647b640 1 -- 172.21.15.1:0/403621848 wait complete. 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 Processor -- start 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 -- start start 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 0x7f6af0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0103c10 0x7f6af0198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af01045d0 0x7f6af01a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010e4c0 con 0x7f6af0102a20 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010e340 con 0x7f6af01045d0 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af647b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010e640 con 0x7f6af0103c10 2024-04-26T22:49:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 0x7f6af0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 0x7f6af0198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46942/0 (socket says 172.21.15.1:46942) 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 -- 172.21.15.1:0/1556857185 learned_addr learned my addr 172.21.15.1:0/1556857185 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aef7fe640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0103c10 0x7f6af0198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 -- 172.21.15.1:0/1556857185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0103c10 msgr2=0x7f6af0198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0103c10 0x7f6af0198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 -- 172.21.15.1:0/1556857185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af01045d0 msgr2=0x7f6af01a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af4a18640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af01045d0 0x7f6af01a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af01045d0 0x7f6af01a0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 -- 172.21.15.1:0/1556857185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af01a06c0 con 0x7f6af0102a20 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aef7fe640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0103c10 0x7f6af0198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6af4a18640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af01045d0 0x7f6af01a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.982+0000 7f6aeffff640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 0x7f6af0198aa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000d770 tx=0x7f6ae000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.983+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0017070 con 0x7f6af0102a20 2024-04-26T22:49:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.983+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ae0009c30 con 0x7f6af0102a20 2024-04-26T22:49:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.983+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af01a09b0 con 0x7f6af0102a20 2024-04-26T22:49:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.983+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0003cc0 con 0x7f6af0102a20 2024-04-26T22:49:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.983+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af01a0f00 con 0x7f6af0102a20 2024-04-26T22:49:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.984+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af0069c30 con 0x7f6af0102a20 2024-04-26T22:49:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.985+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f6ae0003e60 con 0x7f6af0102a20 2024-04-26T22:49:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.985+0000 7f6aed7fa640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6ad80a3c40 0x7f6ad80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.986+0000 7f6aef7fe640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6ad80a3c40 0x7f6ad80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.986+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f6ae00cffe0 con 0x7f6af0102a20 2024-04-26T22:49:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.986+0000 7f6aef7fe640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6ad80a3c40 0x7f6ad80a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6af0199fb0 tx=0x7f6ae4005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:09.990+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ae009c850 con 0x7f6af0102a20 2024-04-26T22:49:10.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.216+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6af002dc10 con 0x7f6af0102a20 2024-04-26T22:49:10.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.217+0000 7f6aed7fa640 1 -- 172.21.15.1:0/1556857185 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11917 (secure 0 0 0) 0x7f6af002dc10 con 0x7f6af0102a20 2024-04-26T22:49:10.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:10.217 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:47:47.801060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[24475,24514,24487,24473,14631]},"id":1}]} 2024-04-26T22:49:10.218 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:49:10.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6ad80a3c40 msgr2=0x7f6ad80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6ad80a3c40 0x7f6ad80a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6af0199fb0 tx=0x7f6ae4005d90 comp rx=0 tx=0).stop 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 msgr2=0x7f6af0198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 0x7f6af0198aa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000d770 tx=0x7f6ae000dc30 comp rx=0 tx=0).stop 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 shutdown_connections 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f6ad80a3c40 0x7f6ad80a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6af0103c10 0x7f6af0198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6af01045d0 0x7f6af01a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 --2- 172.21.15.1:0/1556857185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af0102a20 0x7f6af0198aa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 >> 172.21.15.1:0/1556857185 conn(0x7f6af00fdf90 msgr2=0x7f6af0106eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 shutdown_connections 2024-04-26T22:49:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.219+0000 7f6af647b640 1 -- 172.21.15.1:0/1556857185 wait complete. 2024-04-26T22:49:10.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:49:10.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 -- 172.21.15.1:0/139903021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0102a80 msgr2=0x7f17b0102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 --2- 172.21.15.1:0/139903021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0102a80 0x7f17b0102f00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1798009900 tx=0x7f179802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:10.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 -- 172.21.15.1:0/139903021 shutdown_connections 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 --2- 172.21.15.1:0/139903021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b0103440 0x7f17b0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 --2- 172.21.15.1:0/139903021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0102a80 0x7f17b0102f00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 --2- 172.21.15.1:0/139903021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b0101890 0x7f17b0101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 -- 172.21.15.1:0/139903021 >> 172.21.15.1:0/139903021 conn(0x7f17b00fce40 msgr2=0x7f17b00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 -- 172.21.15.1:0/139903021 shutdown_connections 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 -- 172.21.15.1:0/139903021 wait complete. 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 Processor -- start 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.333+0000 7f17b679d640 1 -- start start 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b679d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b0101890 0x7f17b0198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b679d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b0102a80 0x7f17b01990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b679d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 0x7f17b01a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b679d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17b010e540 con 0x7f17b0103440 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b679d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17b010e3c0 con 0x7f17b0101890 2024-04-26T22:49:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b679d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17b010e6c0 con 0x7f17b0102a80 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 0x7f17b01a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 0x7f17b01a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46964/0 (socket says 172.21.15.1:46964) 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17affff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b0101890 0x7f17b0198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 -- 172.21.15.1:0/1557208183 learned_addr learned my addr 172.21.15.1:0/1557208183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17af7fe640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b0102a80 0x7f17b01990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 -- 172.21.15.1:0/1557208183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b0102a80 msgr2=0x7f17b01990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b0102a80 0x7f17b01990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 -- 172.21.15.1:0/1557208183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b0101890 msgr2=0x7f17b0198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b0101890 0x7f17b0198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 -- 172.21.15.1:0/1557208183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b01a07d0 con 0x7f17b0103440 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.334+0000 7f17b4d3a640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 0x7f17b01a0160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f17a800eaa0 tx=0x7f17a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.335+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a800cdb0 con 0x7f17b0103440 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.335+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17a8005020 con 0x7f17b0103440 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.335+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b01a0ac0 con 0x7f17b0103440 2024-04-26T22:49:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.335+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80116d0 con 0x7f17b0103440 2024-04-26T22:49:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.335+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b01a0fe0 con 0x7f17b0103440 2024-04-26T22:49:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.336+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b0108c50 con 0x7f17b0103440 2024-04-26T22:49:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.337+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f17a80051d0 con 0x7f17b0103440 2024-04-26T22:49:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.337+0000 7f17ad7fa640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f179c0a3cc0 0x7f179c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.338+0000 7f17affff640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f179c0a3cc0 0x7f179c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.338+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f17a80d0640 con 0x7f17b0103440 2024-04-26T22:49:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.338+0000 7f17affff640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f179c0a3cc0 0x7f179c0a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f17b01028e0 tx=0x7f17a000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:10.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.342+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f17a8016030 con 0x7f17b0103440 2024-04-26T22:49:10.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.530+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17b002d110 con 0x7f17b0103440 2024-04-26T22:49:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.532+0000 7f17ad7fa640 1 -- 172.21.15.1:0/1557208183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f17a809c8a0 con 0x7f17b0103440 2024-04-26T22:49:10.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:10.532 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.533+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f179c0a3cc0 msgr2=0x7f179c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.533+0000 7f17b679d640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f179c0a3cc0 0x7f179c0a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f17b01028e0 tx=0x7f17a000a2d0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 msgr2=0x7f17b01a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 0x7f17b01a0160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f17a800eaa0 tx=0x7f17a800ef60 comp rx=0 tx=0).stop 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 shutdown_connections 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f179c0a3cc0 0x7f179c0a6180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b0102a80 0x7f17b01990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17b0103440 0x7f17b01a0160 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 --2- 172.21.15.1:0/1557208183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b0101890 0x7f17b0198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 >> 172.21.15.1:0/1557208183 conn(0x7f17b00fce40 msgr2=0x7f17b0105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:10.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 shutdown_connections 2024-04-26T22:49:10.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.534+0000 7f17b679d640 1 -- 172.21.15.1:0/1557208183 wait complete. 2024-04-26T22:49:10.544 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:10 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3918957370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:10.544 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:10 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/812283228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:10.544 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:10 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1556857185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:10.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:49:10.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 -- 172.21.15.1:0/316682244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0105d40 msgr2=0x7f61f0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 --2- 172.21.15.1:0/316682244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0105d40 0x7f61f0108130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009900 tx=0x7f61e002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 -- 172.21.15.1:0/316682244 shutdown_connections 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 --2- 172.21.15.1:0/316682244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61f0108670 0x7f61f010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 --2- 172.21.15.1:0/316682244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0105d40 0x7f61f0108130 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 --2- 172.21.15.1:0/316682244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0103410 0x7f61f0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 -- 172.21.15.1:0/316682244 >> 172.21.15.1:0/316682244 conn(0x7f61f00fce40 msgr2=0x7f61f00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 -- 172.21.15.1:0/316682244 shutdown_connections 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.647+0000 7f61f7e93640 1 -- 172.21.15.1:0/316682244 wait complete. 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 Processor -- start 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 -- start start 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61f0103410 0x7f61f0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0105d40 0x7f61f01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 0x7f61f01a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f61f010e4d0 con 0x7f61f0108670 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f61f010e350 con 0x7f61f0103410 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f7e93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61f010e650 con 0x7f61f0105d40 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 0x7f61f01a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 0x7f61f01a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46976/0 (socket says 172.21.15.1:46976) 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 -- 172.21.15.1:0/2647932207 learned_addr learned my addr 172.21.15.1:0/2647932207 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f5c2f640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61f0103410 0x7f61f0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f542e640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0105d40 0x7f61f01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 -- 172.21.15.1:0/2647932207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0105d40 msgr2=0x7f61f01991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0105d40 0x7f61f01991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 -- 172.21.15.1:0/2647932207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61f0103410 msgr2=0x7f61f0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61f0103410 0x7f61f0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 -- 172.21.15.1:0/2647932207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f01a0870 con 0x7f61f0108670 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f542e640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0105d40 0x7f61f01991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.648+0000 7f61f6430640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 0x7f61f01a01d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009fd0 tx=0x7f61e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.649+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8017070 con 0x7f61f0108670 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.649+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61e8003e20 con 0x7f61f0108670 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.649+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f01a0b60 con 0x7f61f0108670 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.649+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8010460 con 0x7f61f0108670 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.649+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f01a10b0 con 0x7f61f0108670 2024-04-26T22:49:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.650+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f00fe930 con 0x7f61f0108670 2024-04-26T22:49:10.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.651+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f61e8003970 con 0x7f61f0108670 2024-04-26T22:49:10.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.651+0000 7f61e6ffd640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f61dc0a3cc0 0x7f61dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:10.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.652+0000 7f61f5c2f640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f61dc0a3cc0 0x7f61dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:10.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.652+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f61e80d0400 con 0x7f61f0108670 2024-04-26T22:49:10.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.652+0000 7f61f5c2f640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f61dc0a3cc0 0x7f61dc0a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f61ec005fd0 tx=0x7f61ec005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.655+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f61e809ccd0 con 0x7f61f0108670 2024-04-26T22:49:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:10 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3918957370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:10 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/812283228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:10.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:10 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1556857185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:10.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:10 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3918957370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:10.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:10 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/812283228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:10.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:10 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1556857185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.882+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61f002dad0 con 0x7f61f0108670 2024-04-26T22:49:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.883+0000 7f61e6ffd640 1 -- 172.21.15.1:0/2647932207 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11917 (secure 0 0 0) 0x7f61e809c5d0 con 0x7f61f0108670 2024-04-26T22:49:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:10.883 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:47:47.801060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[24475,24514,24487,24473,14631]},"id":1}]} 2024-04-26T22:49:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f61dc0a3cc0 msgr2=0x7f61dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f61dc0a3cc0 0x7f61dc0a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f61ec005fd0 tx=0x7f61ec005850 comp rx=0 tx=0).stop 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 msgr2=0x7f61f01a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 0x7f61f01a01d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009fd0 tx=0x7f61e800b260 comp rx=0 tx=0).stop 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 shutdown_connections 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f61dc0a3cc0 0x7f61dc0a6180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f0105d40 0x7f61f01991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61f0108670 0x7f61f01a01d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 --2- 172.21.15.1:0/2647932207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61f0103410 0x7f61f0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 >> 172.21.15.1:0/2647932207 conn(0x7f61f00fce40 msgr2=0x7f61f0103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 shutdown_connections 2024-04-26T22:49:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:10.885+0000 7f61f7e93640 1 -- 172.21.15.1:0/2647932207 wait complete. 2024-04-26T22:49:10.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T22:47:41.130042+0000', 'modified': '2024-04-26T22:47:47.801060+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24514, 'mds_1': 24487, 'mds_2': 24475, 'mds_3': 24473, 'mds_4': 14631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.1:6839/1512535471', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6838', 'nonce': 1512535471}, {'type': 'v1', 'addr': '172.21.15.1:6839', 'nonce': 1512535471}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 4, 'incarnation': 23, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6841/1622627716', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6840', 'nonce': 1622627716}, {'type': 'v1', 'addr': '172.21.15.1:6841', 'nonce': 1622627716}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14643': {'gid': 14643, 'name': 'b', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6839/2490292790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 2490292790}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 2490292790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6833/3943944853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6832', 'nonce': 3943944853}, {'type': 'v1', 'addr': '172.21.15.38:6833', 'nonce': 3943944853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6833/4165972801', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6832', 'nonce': 4165972801}, {'type': 'v1', 'addr': '172.21.15.186:6833', 'nonce': 4165972801}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6835/1362256204', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6834', 'nonce': 1362256204}, {'type': 'v1', 'addr': '172.21.15.38:6835', 'nonce': 1362256204}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 3, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.38:6837/1812912662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6836', 'nonce': 1812912662}, {'type': 'v1', 'addr': '172.21.15.38:6837', 'nonce': 1812912662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6835/1379199020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 1379199020}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 1379199020}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.186:6837/786403012', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 786403012}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 786403012}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.186:6839/3709434074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 3709434074}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 3709434074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24514, 'qdb_cluster': [24475, 24514, 24487, 24473, 14631]} 2024-04-26T22:49:10.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-26T22:49:10.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T22:49:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 -- 172.21.15.1:0/1344289608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944102a60 msgr2=0x7f4944102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 --2- 172.21.15.1:0/1344289608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944102a60 0x7f4944102ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4934009900 tx=0x7f493402a410 comp rx=0 tx=0).stop 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 -- 172.21.15.1:0/1344289608 shutdown_connections 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 --2- 172.21.15.1:0/1344289608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4944103420 0x7f4944109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 --2- 172.21.15.1:0/1344289608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944102a60 0x7f4944102ee0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 --2- 172.21.15.1:0/1344289608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4944101870 0x7f4944101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 -- 172.21.15.1:0/1344289608 >> 172.21.15.1:0/1344289608 conn(0x7f49440fce00 msgr2=0x7f49440ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 -- 172.21.15.1:0/1344289608 shutdown_connections 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.000+0000 7f494c247640 1 -- 172.21.15.1:0/1344289608 wait complete. 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 Processor -- start 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 -- start start 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 0x7f494419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4944102a60 0x7f494419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4944103420 0x7f49441a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f494410c2a0 con 0x7f4944101870 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f494410c120 con 0x7f4944102a60 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494c247640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f494410c420 con 0x7f4944103420 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 0x7f494419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 0x7f494419ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46994/0 (socket says 172.21.15.1:46994) 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 -- 172.21.15.1:0/3024463408 learned_addr learned my addr 172.21.15.1:0/3024463408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f49497e2640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4944102a60 0x7f494419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4944103420 0x7f49441a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 -- 172.21.15.1:0/3024463408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4944103420 msgr2=0x7f49441a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4944103420 0x7f49441a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 -- 172.21.15.1:0/3024463408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4944102a60 msgr2=0x7f494419b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4944102a60 0x7f494419b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.001+0000 7f4949fe3640 1 -- 172.21.15.1:0/3024463408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49441a2a60 con 0x7f4944101870 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.002+0000 7f4949fe3640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 0x7f494419ae40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f494000d950 tx=0x7f494000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.002+0000 7f493affd640 1 -- 172.21.15.1:0/3024463408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940017070 con 0x7f4944101870 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.002+0000 7f493affd640 1 -- 172.21.15.1:0/3024463408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4940009c30 con 0x7f4944101870 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.002+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49441a2d50 con 0x7f4944101870 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.002+0000 7f493affd640 1 -- 172.21.15.1:0/3024463408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494000f620 con 0x7f4944101870 2024-04-26T22:49:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.002+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49441a31c0 con 0x7f4944101870 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.004+0000 7f493affd640 1 -- 172.21.15.1:0/3024463408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f494001a070 con 0x7f4944101870 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.004+0000 7f494ef65740 1 Processor -- start 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 -- start start 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f77917ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 0x563f7791cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f7791f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563f7787c740 con 0x563f77918080 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563f778183a0 con 0x563f77917690 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494ef65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f77878700 con 0x563f7791d1f0 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 0x563f7791cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 0x563f7791cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47004/0 (socket says 172.21.15.1:47004) 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 -- 172.21.15.1:0/1621425367 learned_addr learned my addr 172.21.15.1:0/1621425367 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f493affd640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f49300a3c40 0x7f49300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f49497e2640 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f7791f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494a7e4640 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f77917ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f49300a3c40 0x7f49300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f493affd640 1 -- 172.21.15.1:0/3024463408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f49400cff90 con 0x7f4944101870 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 -- 172.21.15.1:0/1621425367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 msgr2=0x563f7791f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f7791f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 -- 172.21.15.1:0/1621425367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 msgr2=0x563f77917ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f77917ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.005+0000 7f4949fe3640 1 -- 172.21.15.1:0/1621425367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f77834750 con 0x563f77918080 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f4949fe3640 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 0x563f7791cb80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f49400cd850 tx=0x7f494000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd1b435c1d99227 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f49300a3c40 0x7f49300a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x563f779188e0 tx=0x7f493c0096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f492ffff640 1 -- 172.21.15.1:0/1621425367 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940021070 con 0x563f77918080 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f492ffff640 1 -- 172.21.15.1:0/1621425367 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f494009c190 con 0x563f77918080 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f492ffff640 1 -- 172.21.15.1:0/1621425367 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49400cdb60 con 0x563f77918080 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 -- 172.21.15.1:0/1621425367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 msgr2=0x563f7791cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 0x563f7791cb80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f49400cd850 tx=0x7f494000bed0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 -- 172.21.15.1:0/1621425367 shutdown_connections 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f7791f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77918080 0x563f7791cb80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 --2- 172.21.15.1:0/1621425367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f77917ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.006+0000 7f494ef65740 1 -- 172.21.15.1:0/1621425367 >> 172.21.15.1:0/1621425367 conn(0x563f77912d00 msgr2=0x563f77913100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 -- 172.21.15.1:0/1621425367 shutdown_connections 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 -- 172.21.15.1:0/1621425367 wait complete. 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 -- start start 2024-04-26T22:49:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 Processor -- start 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77917690 0x563f77921ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77922000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77929040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563f779201a0 con 0x563f7791d1f0 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563f77803fe0 con 0x563f77918080 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f77920320 con 0x563f77917690 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494ef65740 10 client.0 _ll_register_callbacks cb 0x563f7781a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494a7e4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77929040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494a7e4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77929040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47012/0 (socket says 172.21.15.1:47012) 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.007+0000 7f494a7e4640 1 -- 172.21.15.1:0/2869090457 learned_addr learned my addr 172.21.15.1:0/2869090457 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f49497e2640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77922000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f4949fe3640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77917690 0x563f77921ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494a7e4640 1 -- 172.21.15.1:0/2869090457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77917690 msgr2=0x563f77921ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494a7e4640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77917690 0x563f77921ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494a7e4640 1 -- 172.21.15.1:0/2869090457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 msgr2=0x563f77922000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494a7e4640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77922000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494a7e4640 1 -- 172.21.15.1:0/2869090457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f7758e2d0 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f4949fe3640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77917690 0x563f77921ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494a7e4640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77929040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f493c005c20 tx=0x7f493c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d393435f8417bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f492d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f492d7fa640 1 -- 172.21.15.1:0/2869090457 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c004b20 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f492d7fa640 1 -- 172.21.15.1:0/2869090457 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f493c004cc0 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f494009c190 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f492d7fa640 1 -- 172.21.15.1:0/2869090457 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f493c010750 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494ef65740 10 client.15054 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563f776586b0 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.008+0000 7f492d7fa640 1 -- 172.21.15.1:0/2869090457 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f494009c190 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f492d7fa640 1 -- 172.21.15.1:0/2869090457 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f493c005580 con 0x563f7791d1f0 2024-04-26T22:49:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f494ef65740 10 client.15054 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f494ef65740 10 client.15054 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f494ef65740 20 client.15054 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f494ef65740 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f7792e570 0x563f7792f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f494ef65740 4 client.15054 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 --> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f7788a350 con 0x563f7792e570 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f4949fe3640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f7792e570 0x563f7792f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f492d7fa640 10 client.15054 ms_handle_connect on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.009+0000 7f4949fe3640 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f7792e570 0x563f7792f9e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5ae9edbbfc6f516 server_cookie=4348448b6df83cca in_seq=0 out_seq=0 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f492d7fa640 1 -- 172.21.15.1:0/2869090457 <== mds.4 v2:172.21.15.1:6840/1622627716 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f7788a350 con 0x563f7792e570 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f492d7fa640 10 client.15054 handle_command_reply: tid=0 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 client.15054 shutdown 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 msgr2=0x563f77929040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77929040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f493c005c20 tx=0x7f493c004a40 comp rx=0 tx=0).stop 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 shutdown_connections 2024-04-26T22:49:11.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f7792e570 0x563f7792f9e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77929040 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77922000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f492d7fa640 0 client.15054 ms_handle_reset on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 --2- 172.21.15.1:0/2869090457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77917690 0x563f77921ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 >> 172.21.15.1:0/2869090457 conn(0x563f77912d00 msgr2=0x563f77911f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 shutdown_connections 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 1 -- 172.21.15.1:0/2869090457 wait complete. 2024-04-26T22:49:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.010+0000 7f494ef65740 20 client.15054 trim_cache size 0 max 16384 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.028+0000 7f494ef65740 1 Processor -- start 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.028+0000 7f494ef65740 1 -- start start 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494ef65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f779216d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494ef65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77918080 0x563f77928f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494ef65740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77921c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494ef65740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563f7792dd20 con 0x563f7791d1f0 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494ef65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563f7792e3f0 con 0x563f77917690 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494ef65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f7792eac0 con 0x563f77918080 2024-04-26T22:49:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77921c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77921c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47018/0 (socket says 172.21.15.1:47018) 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 -- 172.21.15.1:0/3462016176 learned_addr learned my addr 172.21.15.1:0/3462016176 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 -- 172.21.15.1:0/3462016176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77918080 msgr2=0x563f77928f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77918080 0x563f77928f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 -- 172.21.15.1:0/3462016176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 msgr2=0x563f779216d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f779216d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 -- 172.21.15.1:0/3462016176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f77918f80 con 0x563f7791d1f0 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77921c10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f493c00bb90 tx=0x7f493c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ac256f10ee09b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f492f7fe640 1 -- 172.21.15.1:0/3462016176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c004b20 con 0x563f7791d1f0 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f492f7fe640 1 -- 172.21.15.1:0/3462016176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f493c037020 con 0x563f7791d1f0 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.029+0000 7f492f7fe640 1 -- 172.21.15.1:0/3462016176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c005130 con 0x563f7791d1f0 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 -- 172.21.15.1:0/3462016176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 msgr2=0x563f77921c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77921c10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f493c00bb90 tx=0x7f493c000f40 comp rx=0 tx=0).stop 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 -- 172.21.15.1:0/3462016176 shutdown_connections 2024-04-26T22:49:11.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f77918080 0x563f77928f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f7791d1f0 0x563f77921c10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 --2- 172.21.15.1:0/3462016176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77917690 0x563f779216d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 -- 172.21.15.1:0/3462016176 >> 172.21.15.1:0/3462016176 conn(0x563f77912d00 msgr2=0x563f779194a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 -- 172.21.15.1:0/3462016176 shutdown_connections 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 -- 172.21.15.1:0/3462016176 wait complete. 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 -- start start 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.030+0000 7f494ef65740 1 Processor -- start 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 0x563f77928be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77925680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f77925bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563f7792f860 con 0x563f77917690 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563f7792f190 con 0x563f77918080 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f7792ff30 con 0x563f7791d1f0 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 10 client.0 _ll_register_callbacks cb 0x7f493c012d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 0x563f77928be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 0x563f77928be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47028/0 (socket says 172.21.15.1:47028) 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 -- 172.21.15.1:0/3815685524 learned_addr learned my addr 172.21.15.1:0/3815685524 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494a7e4640 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f77925bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 -- 172.21.15.1:0/3815685524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 msgr2=0x563f77925bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f77925bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 -- 172.21.15.1:0/3815685524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 msgr2=0x563f77925680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77925680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 -- 172.21.15.1:0/3815685524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f77926180 con 0x563f77917690 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f4949fe3640 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 0x563f77928be0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f49400cd940 tx=0x7f494000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733a83b94053be90 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f492d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f492d7fa640 1 -- 172.21.15.1:0/3815685524 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940014020 con 0x563f77917690 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f492d7fa640 1 -- 172.21.15.1:0/3815685524 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f4940098800 con 0x563f77917690 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f493c037020 con 0x563f77917690 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f492d7fa640 1 -- 172.21.15.1:0/3815685524 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4940021040 con 0x563f77917690 2024-04-26T22:49:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 10 client.15060 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.031+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563f77995530 con 0x563f77917690 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f492d7fa640 1 -- 172.21.15.1:0/3815685524 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f493c037020 con 0x563f77917690 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f492d7fa640 1 -- 172.21.15.1:0/3815685524 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f49400d30b0 con 0x563f77917690 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f494ef65740 10 client.15060 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f494ef65740 10 client.15060 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f494ef65740 20 client.15060 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f494ef65740 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f77999c60 0x563f779960a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f494ef65740 4 client.15060 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 --> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] -- command(tid 0: {"prefix": "status"}) -- 0x563f778841c0 con 0x563f77999c60 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f49497e2640 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f77999c60 0x563f779960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f49497e2640 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f77999c60 0x563f779960a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dacce9929ac52e6b server_cookie=a9f2971743732d29 in_seq=0 out_seq=0 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.032+0000 7f492d7fa640 10 client.15060 ms_handle_connect on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f492d7fa640 1 -- 172.21.15.1:0/3815685524 <== mds.4 v2:172.21.15.1:6840/1622627716 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563f778841c0 con 0x563f77999c60 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f492d7fa640 10 client.15060 handle_command_reply: tid=0 2024-04-26T22:49:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 client.15060 shutdown 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 msgr2=0x563f77928be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 0x563f77928be0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f49400cd940 tx=0x7f494000bf60 comp rx=0 tx=0).stop 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 shutdown_connections 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x563f77999c60 0x563f779960a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f7791d1f0 0x563f77925bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f492d7fa640 0 client.15060 ms_handle_reset on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f77918080 0x563f77925680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 --2- 172.21.15.1:0/3815685524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563f77917690 0x563f77928be0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 >> 172.21.15.1:0/3815685524 conn(0x563f77912d00 msgr2=0x563f779216d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 shutdown_connections 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 1 -- 172.21.15.1:0/3815685524 wait complete. 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494ef65740 20 client.15060 trim_cache size 0 max 16384 2024-04-26T22:49:11.039 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 4, 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 14631, 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 84.236422008999995, 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 98.269272306000005 2024-04-26T22:49:11.040 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f49300a3c40 msgr2=0x7f49300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f49300a3c40 0x7f49300a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x563f779188e0 tx=0x7f493c0096a0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 msgr2=0x7f494419ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 0x7f494419ae40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f494000d950 tx=0x7f494000de10 comp rx=0 tx=0).stop 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 shutdown_connections 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f49300a3c40 0x7f49300a6100 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4944103420 0x7f49441a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4944102a60 0x7f494419b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 --2- 172.21.15.1:0/3024463408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4944101870 0x7f494419ae40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.033+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 >> 172.21.15.1:0/3024463408 conn(0x7f49440fce00 msgr2=0x7f4944105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.034+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 shutdown_connections 2024-04-26T22:49:11.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.034+0000 7f494c247640 1 -- 172.21.15.1:0/3024463408 wait complete. 2024-04-26T22:49:11.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 -- 172.21.15.1:0/3142055602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc108660 msgr2=0x7f9bdc10aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/3142055602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc108660 0x7f9bdc10aab0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009900 tx=0x7f9bc402a410 comp rx=0 tx=0).stop 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 -- 172.21.15.1:0/3142055602 shutdown_connections 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/3142055602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc105d30 0x7f9bdc108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/3142055602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc108660 0x7f9bdc10aab0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/3142055602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc103400 0x7f9bdc1057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 -- 172.21.15.1:0/3142055602 >> 172.21.15.1:0/3142055602 conn(0x7f9bdc0fce10 msgr2=0x7f9bdc0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 -- 172.21.15.1:0/3142055602 shutdown_connections 2024-04-26T22:49:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 -- 172.21.15.1:0/3142055602 wait complete. 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 Processor -- start 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.147+0000 7f9be2f0a640 1 -- start start 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 0x7f9bdc198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc105d30 0x7f9bdc199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc108660 0x7f9bdc1a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc10e4b0 con 0x7f9bdc103400 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc10e330 con 0x7f9bdc105d30 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc10e630 con 0x7f9bdc108660 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 0x7f9bdc198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 0x7f9bdc198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47036/0 (socket says 172.21.15.1:47036) 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 -- 172.21.15.1:0/509175078 learned_addr learned my addr 172.21.15.1:0/509175078 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be14a7640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc108660 0x7f9bdc1a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc105d30 0x7f9bdc199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 -- 172.21.15.1:0/509175078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc108660 msgr2=0x7f9bdc1a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc108660 0x7f9bdc1a0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 -- 172.21.15.1:0/509175078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc105d30 msgr2=0x7f9bdc199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc105d30 0x7f9bdc199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 -- 172.21.15.1:0/509175078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc1a0770 con 0x7f9bdc103400 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc105d30 0x7f9bdc199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 0x7f9bdc198b50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400db80 tx=0x7f9bd400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9bd9ffb640 1 -- 172.21.15.1:0/509175078 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4017070 con 0x7f9bdc103400 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc1a0a60 con 0x7f9bdc103400 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9bd9ffb640 1 -- 172.21.15.1:0/509175078 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bd4003b70 con 0x7f9bdc103400 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.148+0000 7f9bd9ffb640 1 -- 172.21.15.1:0/509175078 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd40106c0 con 0x7f9bdc103400 2024-04-26T22:49:11.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.149+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc1a0fb0 con 0x7f9bdc103400 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.150+0000 7f9bd9ffb640 1 -- 172.21.15.1:0/509175078 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9bd4004020 con 0x7f9bdc103400 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 Processor -- start 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 -- start start 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9bd9ffb640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc80a3c40 0x7f9bc80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dc2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 0x558068dc7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9bd9ffb640 1 -- 172.21.15.1:0/509175078 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f9bd40d0620 con 0x7f9bdc103400 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be14a7640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc80a3c40 0x7f9bc80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dcaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558068d23740 con 0x558068dc8600 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558068cbf3a0 con 0x558068dc3490 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be5c28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558068d1f700 con 0x558068dc2aa0 2024-04-26T22:49:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 0x558068dc7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9be14a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dcaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 0x558068dc7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54778/0 (socket says 172.21.15.1:54778) 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.151+0000 7f9bdbfff640 1 -- 172.21.15.1:0/2564676865 learned_addr learned my addr 172.21.15.1:0/2564676865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dc2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bdbfff640 1 -- 172.21.15.1:0/2564676865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 msgr2=0x558068dc2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dc2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bdbfff640 1 -- 172.21.15.1:0/2564676865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 msgr2=0x558068dcaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dcaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bdbfff640 1 -- 172.21.15.1:0/2564676865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558068cdb750 con 0x558068dc3490 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9be14a7640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc80a3c40 0x7f9bc80a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4006010 tx=0x7f9bc4031da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 0x558068dc7f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00dc00 tx=0x7f9bcc00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17180f8c7ce9bd74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bd2ffd640 1 -- 172.21.15.1:0/2564676865 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc005020 con 0x558068dc3490 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bd2ffd640 1 -- 172.21.15.1:0/2564676865 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bcc0047a0 con 0x558068dc3490 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.152+0000 7f9bd2ffd640 1 -- 172.21.15.1:0/2564676865 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0116a0 con 0x558068dc3490 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 -- 172.21.15.1:0/2564676865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 msgr2=0x558068dc7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 0x558068dc7f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00dc00 tx=0x7f9bcc00f730 comp rx=0 tx=0).stop 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 -- 172.21.15.1:0/2564676865 shutdown_connections 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dcaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc3490 0x558068dc7f90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 --2- 172.21.15.1:0/2564676865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dc2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 -- 172.21.15.1:0/2564676865 >> 172.21.15.1:0/2564676865 conn(0x558068db9d00 msgr2=0x558068dba100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 -- 172.21.15.1:0/2564676865 shutdown_connections 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 -- 172.21.15.1:0/2564676865 wait complete. 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.153+0000 7f9be5c28740 1 -- start start 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 Processor -- start 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc2aa0 0x558068dccef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc3490 0x558068dcd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dd4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558068dcb5b0 con 0x558068dc8600 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558068caafe0 con 0x558068dc2aa0 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558068dcb730 con 0x558068dc3490 2024-04-26T22:49:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be5c28740 10 client.0 _ll_register_callbacks cb 0x558068cc1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dd4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dd4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47044/0 (socket says 172.21.15.1:47044) 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 -- 172.21.15.1:0/1224909437 learned_addr learned my addr 172.21.15.1:0/1224909437 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc2aa0 0x558068dccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc3490 0x558068dcd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 -- 172.21.15.1:0/1224909437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc3490 msgr2=0x558068dcd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc3490 0x558068dcd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 -- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc2aa0 msgr2=0x558068dccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc2aa0 0x558068dccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 -- 172.21.15.1:0/1224909437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558068a352d0 con 0x558068dc8600 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9be14a7640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dd4470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4033000 tx=0x7f9bc402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a470563b04a79fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.154+0000 7f9bbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9bbbfff640 1 -- 172.21.15.1:0/1224909437 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4035050 con 0x558068dc8600 2024-04-26T22:49:11.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9bbbfff640 1 -- 172.21.15.1:0/1224909437 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f9bc40037f0 con 0x558068dc8600 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bcc0047a0 con 0x558068dc8600 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9bbbfff640 1 -- 172.21.15.1:0/1224909437 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9bc405a070 con 0x558068dc8600 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 10 client.15072 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558068aff6b0 con 0x558068dc8600 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9bbbfff640 1 -- 172.21.15.1:0/1224909437 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bcc0047a0 con 0x558068dc8600 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9bbbfff640 1 -- 172.21.15.1:0/1224909437 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9bc4051020 con 0x558068dc8600 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 10 client.15072 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 10 client.15072 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 20 client.15072 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068dd9a50 0x558068ddaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 4 client.15072 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.155+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 --> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558068d31350 con 0x558068dd9a50 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.156+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068dd9a50 0x558068ddaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.156+0000 7f9bbbfff640 10 client.15072 ms_handle_connect on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.156+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068dd9a50 0x558068ddaec0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2fb492de315ac267 server_cookie=6645ea33570f7ce4 in_seq=0 out_seq=0 2024-04-26T22:49:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9bbbfff640 1 -- 172.21.15.1:0/1224909437 <== mds.3 v2:172.21.15.38:6836/1812912662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558068d31350 con 0x558068dd9a50 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9bbbfff640 10 client.15072 handle_command_reply: tid=0 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 client.15072 shutdown 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 msgr2=0x558068dd4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dd4470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4033000 tx=0x7f9bc402a410 comp rx=0 tx=0).stop 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 shutdown_connections 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc3490 0x558068dcd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068dd9a50 0x558068ddaec0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc8600 0x558068dd4470 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 --2- 172.21.15.1:0/1224909437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc2aa0 0x558068dccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9bbbfff640 0 client.15072 ms_handle_reset on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 >> 172.21.15.1:0/1224909437 conn(0x558068db9d00 msgr2=0x558068db8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 shutdown_connections 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 1 -- 172.21.15.1:0/1224909437 wait complete. 2024-04-26T22:49:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.157+0000 7f9be5c28740 20 client.15072 trim_cache size 0 max 16384 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 Processor -- start 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 -- start start 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dd8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dd8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 0x558068db8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558068dd9200 con 0x558068dc3490 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558068dd98d0 con 0x558068dc8600 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9be5c28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558068dd9fa0 con 0x558068dc2aa0 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dd8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dd8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47050/0 (socket says 172.21.15.1:47050) 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.175+0000 7f9bdbfff640 1 -- 172.21.15.1:0/580872470 learned_addr learned my addr 172.21.15.1:0/580872470 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bdbfff640 1 -- 172.21.15.1:0/580872470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 msgr2=0x558068dd8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dd8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bdbfff640 1 -- 172.21.15.1:0/580872470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 msgr2=0x558068db8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 0x558068db8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bdbfff640 1 -- 172.21.15.1:0/580872470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558068db9840 con 0x558068dc3490 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dd8bc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00dc00 tx=0x7f9bcc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66adff8b3811b0bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bd27fc640 1 -- 172.21.15.1:0/580872470 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc02a070 con 0x558068dc3490 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bd27fc640 1 -- 172.21.15.1:0/580872470 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bcc005520 con 0x558068dc3490 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9bd27fc640 1 -- 172.21.15.1:0/580872470 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc003e30 con 0x558068dc3490 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9be5c28740 1 -- 172.21.15.1:0/580872470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 msgr2=0x558068dd8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9be5c28740 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dd8bc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00dc00 tx=0x7f9bcc005fd0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9be5c28740 1 -- 172.21.15.1:0/580872470 shutdown_connections 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.176+0000 7f9be5c28740 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 0x558068db8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dd8bc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 --2- 172.21.15.1:0/580872470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dd8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- 172.21.15.1:0/580872470 >> 172.21.15.1:0/580872470 conn(0x558068db9d00 msgr2=0x558068dba0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- 172.21.15.1:0/580872470 shutdown_connections 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- 172.21.15.1:0/580872470 wait complete. 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- start start 2024-04-26T22:49:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 Processor -- start 2024-04-26T22:49:11.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dcf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dcfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 0x558068dd2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558068ddad40 con 0x558068dc3490 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558068dda670 con 0x558068dc8600 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558068ddb410 con 0x558068dc2aa0 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9be5c28740 10 client.0 _ll_register_callbacks cb 0x7f9bc4009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dcfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dcfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47058/0 (socket says 172.21.15.1:47058) 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.177+0000 7f9bdbfff640 1 -- 172.21.15.1:0/724483958 learned_addr learned my addr 172.21.15.1:0/724483958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dcf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bdbfff640 1 -- 172.21.15.1:0/724483958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 msgr2=0x558068dcf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dcf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bdbfff640 1 -- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 msgr2=0x558068dd2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 0x558068dd2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bdbfff640 1 -- 172.21.15.1:0/724483958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558068dd32e0 con 0x558068dc3490 2024-04-26T22:49:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bdbfff640 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dcfcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00df20 tx=0x7f9bcc00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88786a95915ba73e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bbbfff640 1 -- 172.21.15.1:0/724483958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc032020 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bbbfff640 1 -- 172.21.15.1:0/724483958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f9bcc0050c0 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bcc005520 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bbbfff640 1 -- 172.21.15.1:0/724483958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9bcc02a040 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9be5c28740 10 client.15078 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558068e400e0 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bbbfff640 1 -- 172.21.15.1:0/724483958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bcc005520 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9bbbfff640 1 -- 172.21.15.1:0/724483958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9bcc02d070 con 0x558068dc3490 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.178+0000 7f9be5c28740 10 client.15078 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be5c28740 10 client.15078 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be5c28740 20 client.15078 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be5c28740 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068e446f0 0x558068e43400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be5c28740 4 client.15078 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 --> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] -- command(tid 0: {"prefix": "status"}) -- 0x558068d2b1c0 con 0x558068e446f0 2024-04-26T22:49:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068e446f0 0x558068e43400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9bbbfff640 10 client.15078 ms_handle_connect on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.179+0000 7f9be0ca6640 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068e446f0 0x558068e43400 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=443335c1eb50dafc server_cookie=74252b0f17c8deb7 in_seq=0 out_seq=0 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9bbbfff640 1 -- 172.21.15.1:0/724483958 <== mds.3 v2:172.21.15.38:6836/1812912662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558068d2b1c0 con 0x558068e446f0 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9bbbfff640 10 client.15078 handle_command_reply: tid=0 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 client.15078 shutdown 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 msgr2=0x558068dcfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dcfcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00df20 tx=0x7f9bcc00df50 comp rx=0 tx=0).stop 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 shutdown_connections 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x558068e446f0 0x558068e43400 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558068dc8600 0x558068dd2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558068dc3490 0x558068dcfcd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9bbbfff640 0 client.15078 ms_handle_reset on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 --2- 172.21.15.1:0/724483958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558068dc2aa0 0x558068dcf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 >> 172.21.15.1:0/724483958 conn(0x558068db9d00 msgr2=0x558068dc4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 shutdown_connections 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 1 -- 172.21.15.1:0/724483958 wait complete. 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be5c28740 20 client.15078 trim_cache size 0 max 16384 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 3, 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24473, 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 85.356834716999998, 2024-04-26T22:49:11.186 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 106.897025297 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc80a3c40 msgr2=0x7f9bc80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc80a3c40 0x7f9bc80a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4006010 tx=0x7f9bc4031da0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 msgr2=0x7f9bdc198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.180+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 0x7f9bdc198b50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400db80 tx=0x7f9bd400b630 comp rx=0 tx=0).stop 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 shutdown_connections 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f9bc80a3c40 0x7f9bc80a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc108660 0x7f9bdc1a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bdc105d30 0x7f9bdc199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 --2- 172.21.15.1:0/509175078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc103400 0x7f9bdc198b50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 >> 172.21.15.1:0/509175078 conn(0x7f9bdc0fce10 msgr2=0x7f9bdc103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 shutdown_connections 2024-04-26T22:49:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.181+0000 7f9be2f0a640 1 -- 172.21.15.1:0/509175078 wait complete. 2024-04-26T22:49:11.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-26T22:49:11.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 -- 172.21.15.1:0/3798000242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 msgr2=0x7f3cb8102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3798000242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb8102ee0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009900 tx=0x7f3ca802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 -- 172.21.15.1:0/3798000242 shutdown_connections 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3798000242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 0x7f3cb8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3798000242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb8102ee0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3798000242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 0x7f3cb8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 -- 172.21.15.1:0/3798000242 >> 172.21.15.1:0/3798000242 conn(0x7f3cb80fce00 msgr2=0x7f3cb80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 -- 172.21.15.1:0/3798000242 shutdown_connections 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.294+0000 7f3cbf989640 1 -- 172.21.15.1:0/3798000242 wait complete. 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 Processor -- start 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 -- start start 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 0x7f3cb810c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb810a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 0x7f3cb810ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb810e4c0 con 0x7f3cb8102a60 2024-04-26T22:49:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb810e340 con 0x7f3cb8101870 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbf989640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb810e640 con 0x7f3cb8103420 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb810a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb810a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47074/0 (socket says 172.21.15.1:47074) 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3287374154 learned_addr learned my addr 172.21.15.1:0/3287374154 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbd725640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 0x7f3cb810c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 0x7f3cb810ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3287374154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 msgr2=0x7f3cb810ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 0x7f3cb810ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3287374154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 msgr2=0x7f3cb810c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 0x7f3cb810c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3287374154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb810b600 con 0x7f3cb8102a60 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbd725640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 0x7f3cb810c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 0x7f3cb810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.295+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb810a830 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8005940 tx=0x7f3ca8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.296+0000 7f3cae7fc640 1 -- 172.21.15.1:0/3287374154 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca803f030 con 0x7f3cb8102a60 2024-04-26T22:49:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.296+0000 7f3cae7fc640 1 -- 172.21.15.1:0/3287374154 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca8003cb0 con 0x7f3cb8102a60 2024-04-26T22:49:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.296+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb81a2bf0 con 0x7f3cb8102a60 2024-04-26T22:49:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.296+0000 7f3cae7fc640 1 -- 172.21.15.1:0/3287374154 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8035070 con 0x7f3cb8102a60 2024-04-26T22:49:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.296+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb81a3140 con 0x7f3cb8102a60 2024-04-26T22:49:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cae7fc640 1 -- 172.21.15.1:0/3287374154 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3ca8042030 con 0x7f3cb8102a60 2024-04-26T22:49:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 Processor -- start 2024-04-26T22:49:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 -- start start 2024-04-26T22:49:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1d690 0x55b926e1dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1e080 0x55b926e22b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e25680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b926d82740 con 0x55b926e231f0 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b926d1e3a0 con 0x55b926e1e080 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cc2779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b926d7e700 con 0x55b926e1d690 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e25680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e25680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47082/0 (socket says 172.21.15.1:47082) 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3294499597 learned_addr learned my addr 172.21.15.1:0/3294499597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbd725640 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1e080 0x55b926e22b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1d690 0x55b926e1dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3294499597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1d690 msgr2=0x55b926e1dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1d690 0x55b926e1dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3294499597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1e080 msgr2=0x55b926e22b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1e080 0x55b926e22b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3294499597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b926d3a750 con 0x55b926e231f0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.299+0000 7f3cae7fc640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ca40a3c40 0x7f3ca40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.300+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ca40a3c40 0x7f3ca40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.300+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e25680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80ee8a0 tx=0x7f3ca80f92a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79003a8e98099d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.300+0000 7f3cae7fc640 1 -- 172.21.15.1:0/3287374154 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f3ca80f0f80 con 0x7f3cb8102a60 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.300+0000 7f3ca37fe640 1 -- 172.21.15.1:0/3294499597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80bcf00 con 0x55b926e231f0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.300+0000 7f3ca37fe640 1 -- 172.21.15.1:0/3294499597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca80bd830 con 0x55b926e231f0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.300+0000 7f3ca37fe640 1 -- 172.21.15.1:0/3294499597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80bca90 con 0x55b926e231f0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- 172.21.15.1:0/3294499597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 msgr2=0x55b926e25680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ca40a3c40 0x7f3ca40a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55b926e1e8e0 tx=0x7f3cb0009580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e25680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80ee8a0 tx=0x7f3ca80f92a0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- 172.21.15.1:0/3294499597 shutdown_connections 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e25680 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1e080 0x55b926e22b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3294499597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1d690 0x55b926e1dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- 172.21.15.1:0/3294499597 >> 172.21.15.1:0/3294499597 conn(0x55b926e18d00 msgr2=0x55b926e19100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- 172.21.15.1:0/3294499597 shutdown_connections 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- 172.21.15.1:0/3294499597 wait complete. 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- start start 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 Processor -- start 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e27840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1e080 0x55b926e27d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e2edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b926e25f60 con 0x55b926e231f0 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b926d09fe0 con 0x55b926e1d690 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b926e260e0 con 0x55b926e1e080 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.301+0000 7f3cc2779740 10 client.0 _ll_register_callbacks cb 0x55b926d20150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e2edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e2edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47094/0 (socket says 172.21.15.1:47094) 2024-04-26T22:49:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 -- 172.21.15.1:0/1898628127 learned_addr learned my addr 172.21.15.1:0/1898628127 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1e080 0x55b926e27d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbd725640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e27840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 -- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1e080 msgr2=0x55b926e27d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1e080 0x55b926e27d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 -- 172.21.15.1:0/1898628127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 msgr2=0x55b926e27840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e27840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 -- 172.21.15.1:0/1898628127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b926a942d0 con 0x55b926e231f0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e2edc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0010930 tx=0x7f3cb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cc286559b30baf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3ca0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/1898628127 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0004d20 con 0x55b926e231f0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/1898628127 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f3cb000faf0 con 0x55b926e231f0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ca80bd830 con 0x55b926e231f0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/1898628127 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3cb000fcf0 con 0x55b926e231f0 2024-04-26T22:49:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cc2779740 10 client.15090 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.302+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b926b5e6b0 con 0x55b926e231f0 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/1898628127 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca80bd830 con 0x55b926e231f0 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/1898628127 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3cb000f040 con 0x55b926e231f0 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cc2779740 10 client.15090 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cc2779740 10 client.15090 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cc2779740 20 client.15090 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cc2779740 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e34410 0x55b926e35880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cc2779740 4 client.15090 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 --> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b926d90350 con 0x55b926e34410 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.303+0000 7f3cbd725640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e34410 0x55b926e35880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3ca0ff9640 10 client.15090 ms_handle_connect on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cbd725640 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e34410 0x55b926e35880 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4eb25de805c842c server_cookie=ac2109e4a99f874d in_seq=0 out_seq=0 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/1898628127 <== mds.2 v2:172.21.15.186:6834/1379199020 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b926d90350 con 0x55b926e34410 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3ca0ff9640 10 client.15090 handle_command_reply: tid=0 2024-04-26T22:49:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cc2779740 1 client.15090 shutdown 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 msgr2=0x55b926e2edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cc2779740 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e2edc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0010930 tx=0x7f3cb0004a40 comp rx=0 tx=0).stop 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 shutdown_connections 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cc2779740 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e34410 0x55b926e35880 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.304+0000 7f3cc2779740 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e1e080 0x55b926e27d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3ca0ff9640 0 client.15090 ms_handle_reset on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3cc2779740 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e231f0 0x55b926e2edc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3cc2779740 1 --2- 172.21.15.1:0/1898628127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 >> 172.21.15.1:0/1898628127 conn(0x55b926e18d00 msgr2=0x55b926e17f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 shutdown_connections 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3cc2779740 1 -- 172.21.15.1:0/1898628127 wait complete. 2024-04-26T22:49:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.305+0000 7f3cc2779740 20 client.15090 trim_cache size 0 max 16384 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 Processor -- start 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 -- start start 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e26480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b926e33bc0 con 0x55b926e1e080 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b926e34290 con 0x55b926e1d690 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cc2779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b926e34960 con 0x55b926e231f0 2024-04-26T22:49:11.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47106/0 (socket says 172.21.15.1:47106) 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 -- 172.21.15.1:0/2258965874 learned_addr learned my addr 172.21.15.1:0/2258965874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 -- 172.21.15.1:0/2258965874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 msgr2=0x55b926e269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 -- 172.21.15.1:0/2258965874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 msgr2=0x55b926e26480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e26480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 -- 172.21.15.1:0/2258965874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b926e1e910 con 0x55b926e1e080 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.323+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2ece0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80f93d0 tx=0x7f3ca80f4c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c650cca8acb7fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3ca2ffd640 1 -- 172.21.15.1:0/2258965874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8049040 con 0x55b926e1e080 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3ca2ffd640 1 -- 172.21.15.1:0/2258965874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca8003e60 con 0x55b926e1e080 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3ca2ffd640 1 -- 172.21.15.1:0/2258965874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80bcc70 con 0x55b926e1e080 2024-04-26T22:49:11.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 -- 172.21.15.1:0/2258965874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 msgr2=0x55b926e2ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2ece0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80f93d0 tx=0x7f3ca80f4c40 comp rx=0 tx=0).stop 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 -- 172.21.15.1:0/2258965874 shutdown_connections 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2ece0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 --2- 172.21.15.1:0/2258965874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e26480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 -- 172.21.15.1:0/2258965874 >> 172.21.15.1:0/2258965874 conn(0x55b926e18d00 msgr2=0x55b926e27840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 -- 172.21.15.1:0/2258965874 shutdown_connections 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 -- 172.21.15.1:0/2258965874 wait complete. 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.324+0000 7f3cc2779740 1 -- start start 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 Processor -- start 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e2ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e2e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b926e35700 con 0x55b926e1e080 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b926e35030 con 0x55b926e1d690 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b926e35dd0 con 0x55b926e231f0 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cc2779740 10 client.0 _ll_register_callbacks cb 0x7f3cb0009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47108/0 (socket says 172.21.15.1:47108) 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3784118688 learned_addr learned my addr 172.21.15.1:0/3784118688 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbd725640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e2ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbdf26640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e2e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 msgr2=0x55b926e2e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e2e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3784118688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 msgr2=0x55b926e2ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e2ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 -- 172.21.15.1:0/3784118688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b926e2e9a0 con 0x55b926e1e080 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3cbcf24640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2b390 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80b9590 tx=0x7f3ca802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5884f2b81ea3f43b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.325+0000 7f3ca0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/3784118688 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80d6060 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/3784118688 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f3ca80f48a0 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ca8003e60 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/3784118688 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3ca8049070 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 10 client.15096 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b926e9acc0 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/3784118688 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca8003e60 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/3784118688 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3ca80dd070 con 0x55b926e1e080 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 10 client.15096 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 10 client.15096 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 20 client.15096 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e9f6b0 0x55b926e9e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 4 client.15096 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 --> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] -- command(tid 0: {"prefix": "status"}) -- 0x55b926d8a1c0 con 0x55b926e9f6b0 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.326+0000 7f3cbd725640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e9f6b0 0x55b926e9e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cbd725640 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e9f6b0 0x55b926e9e3c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a363bf20c5a595ae server_cookie=e1ece32906bb5ebb in_seq=0 out_seq=0 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3ca0ff9640 10 client.15096 ms_handle_connect on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3ca0ff9640 1 -- 172.21.15.1:0/3784118688 <== mds.2 v2:172.21.15.186:6834/1379199020 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b926d8a1c0 con 0x55b926e9f6b0 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3ca0ff9640 10 client.15096 handle_command_reply: tid=0 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 client.15096 shutdown 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 msgr2=0x55b926e2b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2b390 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80b9590 tx=0x7f3ca802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 shutdown_connections 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x55b926e9f6b0 0x55b926e9e3c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b926e231f0 0x55b926e2e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3ca0ff9640 0 client.15096 ms_handle_reset on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b926e1e080 0x55b926e2b390 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 --2- 172.21.15.1:0/3784118688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b926e1d690 0x55b926e2ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 >> 172.21.15.1:0/3784118688 conn(0x55b926e18d00 msgr2=0x55b926e17670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 shutdown_connections 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 1 -- 172.21.15.1:0/3784118688 wait complete. 2024-04-26T22:49:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.327+0000 7f3cc2779740 20 client.15096 trim_cache size 0 max 16384 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 2, 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24475, 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 86.515405001000005, 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 113.00754305300001 2024-04-26T22:49:11.335 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ca40a3c40 msgr2=0x7f3ca40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ca40a3c40 0x7f3ca40a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55b926e1e8e0 tx=0x7f3cb0009580 comp rx=0 tx=0).stop 2024-04-26T22:49:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 msgr2=0x7f3cb810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb810a830 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8005940 tx=0x7f3ca8004430 comp rx=0 tx=0).stop 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 shutdown_connections 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f3ca40a3c40 0x7f3ca40a6100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb8103420 0x7f3cb810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8102a60 0x7f3cb810a830 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 --2- 172.21.15.1:0/3287374154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb8101870 0x7f3cb810c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 >> 172.21.15.1:0/3287374154 conn(0x7f3cb80fce00 msgr2=0x7f3cb8105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 shutdown_connections 2024-04-26T22:49:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.328+0000 7f3cbf989640 1 -- 172.21.15.1:0/3287374154 wait complete. 2024-04-26T22:49:11.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T22:49:11.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 -- 172.21.15.1:0/425632059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 msgr2=0x7fc608074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/425632059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc608074d40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009900 tx=0x7fc5f802a480 comp rx=0 tx=0).stop 2024-04-26T22:49:11.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 -- 172.21.15.1:0/425632059 shutdown_connections 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/425632059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc60810a830 0x7fc60810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/425632059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc608075310 0x7fc608075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/425632059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc608074d40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 -- 172.21.15.1:0/425632059 >> 172.21.15.1:0/425632059 conn(0x7fc6080fdef0 msgr2=0x7fc608100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 -- 172.21.15.1:0/425632059 shutdown_connections 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.442+0000 7fc60e6ea640 1 -- 172.21.15.1:0/425632059 wait complete. 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 Processor -- start 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 -- start start 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc608075310 0x7fc60819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc60819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc60810a830 0x7fc6081a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6081106b0 con 0x7fc6080768f0 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc608110530 con 0x7fc608075310 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc60e6ea640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc608110830 con 0x7fc60810a830 2024-04-26T22:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc60819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc60819b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47128/0 (socket says 172.21.15.1:47128) 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 -- 172.21.15.1:0/1215518161 learned_addr learned my addr 172.21.15.1:0/1215518161 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc607fff640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc608075310 0x7fc60819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 -- 172.21.15.1:0/1215518161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc60810a830 msgr2=0x7fc6081a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc60810a830 0x7fc6081a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 -- 172.21.15.1:0/1215518161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc608075310 msgr2=0x7fc60819adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc608075310 0x7fc60819adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.443+0000 7fc6077fe640 1 -- 172.21.15.1:0/1215518161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6081a29d0 con 0x7fc6080768f0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.444+0000 7fc6077fe640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc60819b2f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00d950 tx=0x7fc5fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.444+0000 7fc6057fa640 1 -- 172.21.15.1:0/1215518161 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc017070 con 0x7fc6080768f0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.444+0000 7fc6057fa640 1 -- 172.21.15.1:0/1215518161 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5fc009c30 con 0x7fc6080768f0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.444+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6081a2cc0 con 0x7fc6080768f0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.444+0000 7fc6057fa640 1 -- 172.21.15.1:0/1215518161 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00f620 con 0x7fc6080768f0 2024-04-26T22:49:11.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.444+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6081a31c0 con 0x7fc6080768f0 2024-04-26T22:49:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.446+0000 7fc6057fa640 1 -- 172.21.15.1:0/1215518161 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc5fc010460 con 0x7fc6080768f0 2024-04-26T22:49:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.446+0000 7fc611400740 1 Processor -- start 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.446+0000 7fc611400740 1 -- start start 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.446+0000 7fc611400740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc95ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc611400740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc611400740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc9d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc611400740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdbfa740 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc6057fa640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5f00a3c40 0x7fc5f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc611400740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdb963a0 con 0x55e3bdc95690 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc611400740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdbf6700 con 0x55e3bdc96080 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc9d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc6077fe640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5f00a3c40 0x7fc5f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc9d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47130/0 (socket says 172.21.15.1:47130) 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 -- 172.21.15.1:0/3511108355 learned_addr learned my addr 172.21.15.1:0/3511108355 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc95ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc607fff640 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc6057fa640 1 -- 172.21.15.1:0/1215518161 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc5fc0d0210 con 0x7fc6080768f0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 -- 172.21.15.1:0/3511108355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc96080 msgr2=0x55e3bdc9ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 -- 172.21.15.1:0/3511108355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 msgr2=0x55e3bdc95ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc95ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 -- 172.21.15.1:0/3511108355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3bdbb2750 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc6077fe640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5f00a3c40 0x7fc5f00a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc09c2f0 tx=0x7fc5fc0c7470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc9d680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc600012a10 tx=0x7fc6000096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c668bf50f452f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc5ee7fc640 1 -- 172.21.15.1:0/3511108355 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60000d9a0 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc5ee7fc640 1 -- 172.21.15.1:0/3511108355 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6000103d0 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.447+0000 7fc5ee7fc640 1 -- 172.21.15.1:0/3511108355 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60000f5c0 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 -- 172.21.15.1:0/3511108355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 msgr2=0x55e3bdc9d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc9d680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc600012a10 tx=0x7fc6000096c0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 -- 172.21.15.1:0/3511108355 shutdown_connections 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc9d680 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 --2- 172.21.15.1:0/3511108355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc95ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 -- 172.21.15.1:0/3511108355 >> 172.21.15.1:0/3511108355 conn(0x55e3bdc90d00 msgr2=0x55e3bdc91100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 -- 172.21.15.1:0/3511108355 shutdown_connections 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 -- 172.21.15.1:0/3511108355 wait complete. 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.448+0000 7fc611400740 1 -- start start 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 Processor -- start 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc9fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdc9e1a0 con 0x55e3bdc96080 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdb81fe0 con 0x55e3bdc95690 2024-04-26T22:49:11.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdc9e320 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc611400740 10 client.0 _ll_register_callbacks cb 0x55e3bdb98150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47140/0 (socket says 172.21.15.1:47140) 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 -- 172.21.15.1:0/1847420579 learned_addr learned my addr 172.21.15.1:0/1847420579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc60cc87640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc9fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc60cc87640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 -- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 msgr2=0x55e3bdca6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 -- 172.21.15.1:0/1847420579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 msgr2=0x55e3bdc9fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc9fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc607fff640 1 -- 172.21.15.1:0/1847420579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3bd90c2d0 con 0x55e3bdc96080 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.449+0000 7fc60cc87640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc607fff640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ffa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc5f802a9a0 tx=0x7fc5f8031f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee2ac09cb913700 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc5e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc5e77fe640 1 -- 172.21.15.1:0/1847420579 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8035040 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc5e77fe640 1 -- 172.21.15.1:0/1847420579 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc5f8004040 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6000103d0 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc5e77fe640 1 -- 172.21.15.1:0/1847420579 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc5f804d070 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 10 client.15108 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e3bd9d66b0 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc5e77fe640 1 -- 172.21.15.1:0/1847420579 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6000103d0 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc5e77fe640 1 -- 172.21.15.1:0/1847420579 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc5f8044020 con 0x55e3bdc96080 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 10 client.15108 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 10 client.15108 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 20 client.15108 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdcac550 0x55e3bdcad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 4 client.15108 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.450+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 --> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3bdc08350 con 0x55e3bdcac550 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.451+0000 7fc60cc87640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdcac550 0x55e3bdcad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.451+0000 7fc60cc87640 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdcac550 0x55e3bdcad9c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7dfea847bbf05b9 server_cookie=b2bae48d4cfd3ded in_seq=0 out_seq=0 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.451+0000 7fc5e77fe640 10 client.15108 ms_handle_connect on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc5e77fe640 1 -- 172.21.15.1:0/1847420579 <== mds.1 v2:172.21.15.186:6836/786403012 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3bdc08350 con 0x55e3bdcac550 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc5e77fe640 10 client.15108 handle_command_reply: tid=0 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 client.15108 shutdown 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 msgr2=0x55e3bdc9ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ffa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc5f802a9a0 tx=0x7fc5f8031f10 comp rx=0 tx=0).stop 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 shutdown_connections 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdcac550 0x55e3bdcad9c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc5e77fe640 0 client.15108 ms_handle_reset on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdc9ffa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 --2- 172.21.15.1:0/1847420579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdc9fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 >> 172.21.15.1:0/1847420579 conn(0x55e3bdc90d00 msgr2=0x55e3bdc8ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 shutdown_connections 2024-04-26T22:49:11.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 1 -- 172.21.15.1:0/1847420579 wait complete. 2024-04-26T22:49:11.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.452+0000 7fc611400740 20 client.15108 trim_cache size 0 max 16384 2024-04-26T22:49:11.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 Processor -- start 2024-04-26T22:49:11.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 -- start start 2024-04-26T22:49:11.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc95690 0x55e3bdc97440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc97980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdcabd00 con 0x55e3bdc96080 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdcac3d0 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc611400740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdcacaa0 con 0x55e3bdc95690 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47146/0 (socket says 172.21.15.1:47146) 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc607fff640 1 -- 172.21.15.1:0/3346238813 learned_addr learned my addr 172.21.15.1:0/3346238813 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc97980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.471+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc95690 0x55e3bdc97440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc607fff640 1 -- 172.21.15.1:0/3346238813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc95690 msgr2=0x55e3bdc97440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc607fff640 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc95690 0x55e3bdc97440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc607fff640 1 -- 172.21.15.1:0/3346238813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc9b1f0 msgr2=0x55e3bdc97980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc607fff640 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc97980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc607fff640 1 -- 172.21.15.1:0/3346238813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3bdc96460 con 0x55e3bdc96080 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc60cc87640 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc95690 0x55e3bdc97440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc607fff640 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca6fa0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc5f802a4c0 tx=0x7fc5f8005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c934260258896173 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc5edffb640 1 -- 172.21.15.1:0/3346238813 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8057070 con 0x55e3bdc96080 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc5edffb640 1 -- 172.21.15.1:0/3346238813 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5f8046830 con 0x55e3bdc96080 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.472+0000 7fc5edffb640 1 -- 172.21.15.1:0/3346238813 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8035070 con 0x55e3bdc96080 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 -- 172.21.15.1:0/3346238813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 msgr2=0x55e3bdca6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca6fa0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc5f802a4c0 tx=0x7fc5f8005a60 comp rx=0 tx=0).stop 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 -- 172.21.15.1:0/3346238813 shutdown_connections 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdc97980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca6fa0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 --2- 172.21.15.1:0/3346238813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc95690 0x55e3bdc97440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 -- 172.21.15.1:0/3346238813 >> 172.21.15.1:0/3346238813 conn(0x55e3bdc90d00 msgr2=0x55e3bdcab370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 -- 172.21.15.1:0/3346238813 shutdown_connections 2024-04-26T22:49:11.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 -- 172.21.15.1:0/3346238813 wait complete. 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 -- start start 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.473+0000 7fc611400740 1 Processor -- start 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdca2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdcad840 con 0x55e3bdc96080 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdcad170 con 0x55e3bdc95690 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e3bdcadf10 con 0x55e3bdc9b1f0 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 10 client.0 _ll_register_callbacks cb 0x7fc5f8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47154/0 (socket says 172.21.15.1:47154) 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 -- 172.21.15.1:0/272154895 learned_addr learned my addr 172.21.15.1:0/272154895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc60cc87640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdca2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc60cc87640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 -- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 msgr2=0x55e3bdca6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 -- 172.21.15.1:0/272154895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 msgr2=0x55e3bdca2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdca2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 -- 172.21.15.1:0/272154895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3bdca6800 con 0x55e3bdc96080 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc607fff640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca31f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8055040 tx=0x7fc5f8004210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d57da219004e3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc5e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc5e77fe640 1 -- 172.21.15.1:0/272154895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8035070 con 0x55e3bdc96080 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc5e77fe640 1 -- 172.21.15.1:0/272154895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc5f8046c90 con 0x55e3bdc96080 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.474+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5f8046830 con 0x55e3bdc96080 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc5e77fe640 1 -- 172.21.15.1:0/272154895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc5f8057070 con 0x55e3bdc96080 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 10 client.15114 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e3bdd12a30 con 0x55e3bdc96080 2024-04-26T22:49:11.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc5e77fe640 1 -- 172.21.15.1:0/272154895 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5f8046830 con 0x55e3bdc96080 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc5e77fe640 1 -- 172.21.15.1:0/272154895 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc5f8044070 con 0x55e3bdc96080 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 10 client.15114 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 10 client.15114 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 20 client.15114 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdd17240 0x55e3bdd15f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 4 client.15114 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 --> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] -- command(tid 0: {"prefix": "status"}) -- 0x55e3bdc021c0 con 0x55e3bdd17240 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.475+0000 7fc60cc87640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdd17240 0x55e3bdd15f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc60cc87640 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdd17240 0x55e3bdd15f50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dff5c2ef7fce5df2 server_cookie=8d0ca54c3e400831 in_seq=0 out_seq=0 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc5e77fe640 10 client.15114 ms_handle_connect on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc5e77fe640 1 -- 172.21.15.1:0/272154895 <== mds.1 v2:172.21.15.186:6836/786403012 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e3bdc021c0 con 0x55e3bdd17240 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc5e77fe640 10 client.15114 handle_command_reply: tid=0 2024-04-26T22:49:11.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 client.15114 shutdown 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 msgr2=0x55e3bdca31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca31f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8055040 tx=0x7fc5f8004210 comp rx=0 tx=0).stop 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 shutdown_connections 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x55e3bdd17240 0x55e3bdd15f50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3bdc9b1f0 0x55e3bdca6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e3bdc96080 0x55e3bdca31f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc5e77fe640 0 client.15114 ms_handle_reset on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 --2- 172.21.15.1:0/272154895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e3bdc95690 0x55e3bdca2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 >> 172.21.15.1:0/272154895 conn(0x55e3bdc90d00 msgr2=0x55e3bdcab370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 shutdown_connections 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 1 -- 172.21.15.1:0/272154895 wait complete. 2024-04-26T22:49:11.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.476+0000 7fc611400740 20 client.15114 trim_cache size 0 max 16384 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24487, 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 87.668766536000007, 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 110.289355839 2024-04-26T22:49:11.484 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:11.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5f00a3c40 msgr2=0x7fc5f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5f00a3c40 0x7fc5f00a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc09c2f0 tx=0x7fc5fc0c7470 comp rx=0 tx=0).stop 2024-04-26T22:49:11.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 msgr2=0x7fc60819b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc60819b2f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00d950 tx=0x7fc5fc00de10 comp rx=0 tx=0).stop 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 shutdown_connections 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc5f00a3c40 0x7fc5f00a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc60810a830 0x7fc6081a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6080768f0 0x7fc60819b2f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 --2- 172.21.15.1:0/1215518161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc608075310 0x7fc60819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 >> 172.21.15.1:0/1215518161 conn(0x7fc6080fdef0 msgr2=0x7fc6081067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 shutdown_connections 2024-04-26T22:49:11.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.477+0000 7fc60e6ea640 1 -- 172.21.15.1:0/1215518161 wait complete. 2024-04-26T22:49:11.490 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T22:49:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- 172.21.15.1:0/4148918615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64105d30 msgr2=0x7f1a64108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- 172.21.15.1:0/4148918615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64105d30 0x7f1a64108120 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009900 tx=0x7f1a5802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- 172.21.15.1:0/4148918615 shutdown_connections 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- 172.21.15.1:0/4148918615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a64108660 0x7f1a6410aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- 172.21.15.1:0/4148918615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64105d30 0x7f1a64108120 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- 172.21.15.1:0/4148918615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64103400 0x7f1a641057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- 172.21.15.1:0/4148918615 >> 172.21.15.1:0/4148918615 conn(0x7f1a640fce10 msgr2=0x7f1a640ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- 172.21.15.1:0/4148918615 shutdown_connections 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- 172.21.15.1:0/4148918615 wait complete. 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 Processor -- start 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- start start 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 0x7f1a64198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64105d30 0x7f1a64199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a64108660 0x7f1a641a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6410e4c0 con 0x7f1a64103400 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6410e340 con 0x7f1a64108660 2024-04-26T22:49:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a6abda640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6410e640 con 0x7f1a64105d30 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a68976640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 0x7f1a64198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a68976640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 0x7f1a64198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47178/0 (socket says 172.21.15.1:47178) 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.593+0000 7f1a68976640 1 -- 172.21.15.1:0/1200032678 learned_addr learned my addr 172.21.15.1:0/1200032678 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a63fff640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64105d30 0x7f1a64199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a68976640 1 -- 172.21.15.1:0/1200032678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64105d30 msgr2=0x7f1a64199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a68976640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64105d30 0x7f1a64199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a68976640 1 -- 172.21.15.1:0/1200032678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a64108660 msgr2=0x7f1a641a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a69177640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a64108660 0x7f1a641a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a68976640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a64108660 0x7f1a641a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a68976640 1 -- 172.21.15.1:0/1200032678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a641a0860 con 0x7f1a64103400 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a63fff640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64105d30 0x7f1a64199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:11.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a68976640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 0x7f1a64198c50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00eb90 tx=0x7f1a5c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a61ffb640 1 -- 172.21.15.1:0/1200032678 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c004d20 con 0x7f1a64103400 2024-04-26T22:49:11.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a641a0b50 con 0x7f1a64103400 2024-04-26T22:49:11.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a61ffb640 1 -- 172.21.15.1:0/1200032678 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a5c00ee90 con 0x7f1a64103400 2024-04-26T22:49:11.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a61ffb640 1 -- 172.21.15.1:0/1200032678 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c010670 con 0x7f1a64103400 2024-04-26T22:49:11.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.594+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a641a10a0 con 0x7f1a64103400 2024-04-26T22:49:11.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.596+0000 7f1a61ffb640 1 -- 172.21.15.1:0/1200032678 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1a5c005020 con 0x7f1a64103400 2024-04-26T22:49:11.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 Processor -- start 2024-04-26T22:49:11.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 -- start start 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4abec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a61ffb640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1a4c0a3c40 0x7f1a4c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4ac490 0x559f3c4b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559f3c40c740 con 0x559f3c4abaa0 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f3c3a83a0 con 0x559f3c4ac490 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f3c408700 con 0x559f3c4b1600 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a68976640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1a4c0a3c40 0x7f1a4c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4ac490 0x559f3c4b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4abec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47190/0 (socket says 172.21.15.1:47190) 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 -- 172.21.15.1:0/41951734 learned_addr learned my addr 172.21.15.1:0/41951734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a61ffb640 1 -- 172.21.15.1:0/1200032678 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f1a5c0d0db0 con 0x7f1a64103400 2024-04-26T22:49:11.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 -- 172.21.15.1:0/41951734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 msgr2=0x559f3c4b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 -- 172.21.15.1:0/41951734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4ac490 msgr2=0x559f3c4b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4ac490 0x559f3c4b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.597+0000 7f1a69177640 1 -- 172.21.15.1:0/41951734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f3c3c4750 con 0x559f3c4abaa0 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a68976640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1a4c0a3c40 0x7f1a4c0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c09ce90 tx=0x7f1a5c0879f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a69177640 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4abec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400bac0 tx=0x7f1a5400bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51138032ea3e8181 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a4affd640 1 -- 172.21.15.1:0/41951734 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54005020 con 0x559f3c4abaa0 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a4affd640 1 -- 172.21.15.1:0/41951734 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a540047a0 con 0x559f3c4abaa0 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a4affd640 1 -- 172.21.15.1:0/41951734 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a540116d0 con 0x559f3c4abaa0 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/41951734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 msgr2=0x559f3c4abec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.598+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4abec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400bac0 tx=0x7f1a5400bf80 comp rx=0 tx=0).stop 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/41951734 shutdown_connections 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4ac490 0x559f3c4b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/41951734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4abec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/41951734 >> 172.21.15.1:0/41951734 conn(0x559f3c4a2d00 msgr2=0x559f3c4a3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/41951734 shutdown_connections 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/41951734 wait complete. 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- start start 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 Processor -- start 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4abaa0 0x559f3c4b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4bd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4b45b0 con 0x559f3c4ac490 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f3c393fe0 con 0x559f3c4abaa0 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4b4730 con 0x559f3c4b1600 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a6d9c9740 10 client.0 _ll_register_callbacks cb 0x559f3c3aa150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47198/0 (socket says 172.21.15.1:47198) 2024-04-26T22:49:11.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a63fff640 1 -- 172.21.15.1:0/2081102456 learned_addr learned my addr 172.21.15.1:0/2081102456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.599+0000 7f1a69177640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4abaa0 0x559f3c4b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a69177640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4bd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a63fff640 1 -- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 msgr2=0x559f3c4bd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a63fff640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4bd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a63fff640 1 -- 172.21.15.1:0/2081102456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4abaa0 msgr2=0x559f3c4b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a63fff640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4abaa0 0x559f3c4b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a63fff640 1 -- 172.21.15.1:0/2081102456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3c11e2d0 con 0x559f3c4ac490 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a63fff640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4b63d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1a58005c40 tx=0x7f1a58032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b389988d4efe75d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a43fff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a43fff640 1 -- 172.21.15.1:0/2081102456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58040070 con 0x559f3c4ac490 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a540047a0 con 0x559f3c4ac490 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a43fff640 1 -- 172.21.15.1:0/2081102456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f1a58004240 con 0x559f3c4ac490 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a43fff640 1 -- 172.21.15.1:0/2081102456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1a58035040 con 0x559f3c4ac490 2024-04-26T22:49:11.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a6d9c9740 10 client.15126 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f3c1e86b0 con 0x559f3c4ac490 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.600+0000 7f1a43fff640 1 -- 172.21.15.1:0/2081102456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a540047a0 con 0x559f3c4ac490 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a43fff640 1 -- 172.21.15.1:0/2081102456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1a580049f0 con 0x559f3c4ac490 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a6d9c9740 10 client.15126 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a6d9c9740 10 client.15126 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a6d9c9740 20 client.15126 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c4c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a6d9c9740 4 client.15126 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 --> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f3c41a350 con 0x559f3c4c2990 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a69177640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c4c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a69177640 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c4c3e00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9167e6ed33366833 server_cookie=492da01a801f28d2 in_seq=0 out_seq=0 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.601+0000 7f1a43fff640 10 client.15126 ms_handle_connect on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a43fff640 1 -- 172.21.15.1:0/2081102456 <== mds.0 v2:172.21.15.186:6838/3709434074 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f3c41a350 con 0x559f3c4c2990 2024-04-26T22:49:11.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a43fff640 10 client.15126 handle_command_reply: tid=0 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 client.15126 shutdown 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 msgr2=0x559f3c4b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4b63d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1a58005c40 tx=0x7f1a58032f20 comp rx=0 tx=0).stop 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 shutdown_connections 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c4c3e00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4b1600 0x559f3c4bd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a43fff640 0 client.15126 ms_handle_reset on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4b63d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/2081102456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4abaa0 0x559f3c4b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.602+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 >> 172.21.15.1:0/2081102456 conn(0x559f3c4a2d00 msgr2=0x559f3c4a1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.603+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 shutdown_connections 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.603+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/2081102456 wait complete. 2024-04-26T22:49:11.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.603+0000 7f1a6d9c9740 20 client.15126 trim_cache size 0 max 16384 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 Processor -- start 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 -- start start 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4acd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4ac490 0x559f3c4bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4ad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4c2140 con 0x559f3c4abaa0 2024-04-26T22:49:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4c2810 con 0x559f3c4b1600 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4c2ee0 con 0x559f3c4ac490 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4acd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47206/0 (socket says 172.21.15.1:47206) 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 -- 172.21.15.1:0/1116560491 learned_addr learned my addr 172.21.15.1:0/1116560491 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 -- 172.21.15.1:0/1116560491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4ac490 msgr2=0x559f3c4bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4ac490 0x559f3c4bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 -- 172.21.15.1:0/1116560491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 msgr2=0x559f3c4ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 -- 172.21.15.1:0/1116560491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f3c4c0300 con 0x559f3c4abaa0 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.621+0000 7f1a69177640 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4acd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400bac0 tx=0x7f1a54005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4ebccecf60eee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a4a7fc640 1 -- 172.21.15.1:0/1116560491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54024040 con 0x559f3c4abaa0 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a4a7fc640 1 -- 172.21.15.1:0/1116560491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a54004880 con 0x559f3c4abaa0 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a4a7fc640 1 -- 172.21.15.1:0/1116560491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5402c380 con 0x559f3c4abaa0 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1116560491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 msgr2=0x559f3c4acd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4acd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400bac0 tx=0x7f1a54005d40 comp rx=0 tx=0).stop 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1116560491 shutdown_connections 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4ac490 0x559f3c4bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1116560491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4abaa0 0x559f3c4acd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1116560491 >> 172.21.15.1:0/1116560491 conn(0x559f3c4a2d00 msgr2=0x559f3c4a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1116560491 shutdown_connections 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1116560491 wait complete. 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.622+0000 7f1a6d9c9740 1 -- start start 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 Processor -- start 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4abaa0 0x559f3c4bb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4bbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4bc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4c3c80 con 0x559f3c4ac490 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4c35b0 con 0x559f3c4b1600 2024-04-26T22:49:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f3c4c4350 con 0x559f3c4abaa0 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a6d9c9740 10 client.0 _ll_register_callbacks cb 0x7f1a58009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4bbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4bbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47214/0 (socket says 172.21.15.1:47214) 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 -- 172.21.15.1:0/1662738368 learned_addr learned my addr 172.21.15.1:0/1662738368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a69177640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 -- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4abaa0 msgr2=0x559f3c4bb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4abaa0 0x559f3c4bb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 -- 172.21.15.1:0/1662738368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 msgr2=0x559f3c4bc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4bc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 -- 172.21.15.1:0/1662738368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3c4bc8c0 con 0x559f3c4ac490 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a69177640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a63fff640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4bbb50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1a58005940 tx=0x7f1a58031ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50414c4e9ced931c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.623+0000 7f1a43fff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:11.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a43fff640 1 -- 172.21.15.1:0/1662738368 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58040070 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a43fff640 1 -- 172.21.15.1:0/1662738368 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f1a58004ad0 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559f3c4141c0 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a43fff640 1 -- 172.21.15.1:0/1662738368 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1a58009520 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 10 client.15132 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f3c528b80 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a43fff640 1 -- 172.21.15.1:0/1662738368 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559f3c4141c0 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a43fff640 1 -- 172.21.15.1:0/1662738368 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1a58057060 con 0x559f3c4ac490 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 10 client.15132 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 10 client.15132 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 20 client.15132 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c52c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 4 client.15132 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 --> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] -- command(tid 0: {"prefix": "status"}) -- 0x559f3c38f9a0 con 0x559f3c4c2990 2024-04-26T22:49:11.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.624+0000 7f1a69177640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c52c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a69177640 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c52c4e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5a728862c11a3f7 server_cookie=8aea984633be3045 in_seq=0 out_seq=0 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a43fff640 10 client.15132 ms_handle_connect on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a43fff640 1 -- 172.21.15.1:0/1662738368 <== mds.0 v2:172.21.15.186:6838/3709434074 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x559f3c38f9a0 con 0x559f3c4c2990 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a43fff640 10 client.15132 handle_command_reply: tid=0 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 client.15132 shutdown 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 msgr2=0x559f3c4bbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4bbb50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1a58005940 tx=0x7f1a58031ca0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 shutdown_connections 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x559f3c4c2990 0x559f3c52c4e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f3c4b1600 0x559f3c4bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f3c4ac490 0x559f3c4bbb50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 --2- 172.21.15.1:0/1662738368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f3c4abaa0 0x559f3c4bb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a43fff640 0 client.15132 ms_handle_reset on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 >> 172.21.15.1:0/1662738368 conn(0x559f3c4a2d00 msgr2=0x559f3c4acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 shutdown_connections 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 1 -- 172.21.15.1:0/1662738368 wait complete. 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.625+0000 7f1a6d9c9740 20 client.15132 trim_cache size 0 max 16384 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24514, 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 89.859329613, 2024-04-26T22:49:11.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:11.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:11.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:11.633 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 104.471454991 2024-04-26T22:49:11.633 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1a4c0a3c40 msgr2=0x7f1a4c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1a4c0a3c40 0x7f1a4c0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c09ce90 tx=0x7f1a5c0879f0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 msgr2=0x7f1a64198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 0x7f1a64198c50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00eb90 tx=0x7f1a5c00c6a0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 shutdown_connections 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1a4c0a3c40 0x7f1a4c0a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a64105d30 0x7f1a64199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a64108660 0x7f1a641a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 --2- 172.21.15.1:0/1200032678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a64103400 0x7f1a64198c50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 >> 172.21.15.1:0/1200032678 conn(0x7f1a640fce10 msgr2=0x7f1a64103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 shutdown_connections 2024-04-26T22:49:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:11.626+0000 7f1a6abda640 1 -- 172.21.15.1:0/1200032678 wait complete. 2024-04-26T22:49:11.637 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T22:49:11.637 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T22:49:11.648 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T22:49:11.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T22:49:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:11 smithi186 ceph-mon[32774]: pgmap v337: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T22:49:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:11 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1557208183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:11 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2647932207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:11 smithi001 ceph-mon[29435]: pgmap v337: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T22:49:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:11 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1557208183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:11 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2647932207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:11 smithi038 ceph-mon[32433]: pgmap v337: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T22:49:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:11 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1557208183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:11 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2647932207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:11.893 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:13.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:13 smithi001 ceph-mon[29435]: pgmap v338: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T22:49:13.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 -- 172.21.15.1:0/1968656876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090106840 msgr2=0x7f9090108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1968656876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090106840 0x7f9090108c30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9084009990 tx=0x7f9084009ca0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 -- 172.21.15.1:0/1968656876 shutdown_connections 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1968656876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9090103f10 0x7f9090106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1968656876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9090109170 0x7f909010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1968656876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090106840 0x7f9090108c30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 -- 172.21.15.1:0/1968656876 >> 172.21.15.1:0/1968656876 conn(0x7f90900fd940 msgr2=0x7f90900ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 -- 172.21.15.1:0/1968656876 shutdown_connections 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 -- 172.21.15.1:0/1968656876 wait complete. 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.699+0000 7f9095ea3700 1 Processor -- start 2024-04-26T22:49:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 -- start start 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 0x7f9090103870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9090106840 0x7f9090101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9090109170 0x7f9090102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f909010ef40 con 0x7f9090103f10 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f909010edc0 con 0x7f9090106840 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f9095ea3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f909010f0c0 con 0x7f9090109170 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 0x7f9090103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 0x7f9090103870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43684/0 (socket says 172.21.15.1:43684) 2024-04-26T22:49:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 -- 172.21.15.1:0/1224574496 learned_addr learned my addr 172.21.15.1:0/1224574496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908ffff700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9090109170 0x7f9090102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908effd700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9090106840 0x7f9090101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 -- 172.21.15.1:0/1224574496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9090109170 msgr2=0x7f9090102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9090109170 0x7f9090102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 -- 172.21.15.1:0/1224574496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9090106840 msgr2=0x7f9090101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9090106840 0x7f9090101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 -- 172.21.15.1:0/1224574496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9090102cb0 con 0x7f9090103f10 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.700+0000 7f908f7fe700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 0x7f9090103870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f908000d980 tx=0x7f908000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.701+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9080017070 con 0x7f9090103f10 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.701+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9080005020 con 0x7f9090103f10 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.701+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908000fa10 con 0x7f9090103f10 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.701+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90901a3780 con 0x7f9090103f10 2024-04-26T22:49:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.701+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90901a3c90 con 0x7f9090103f10 2024-04-26T22:49:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.702+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f909006a110 con 0x7f9090103f10 2024-04-26T22:49:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.703+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9080004a30 con 0x7f9090103f10 2024-04-26T22:49:13.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.704+0000 7f908cff9700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f90780a3d50 0x7f90780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.704+0000 7f908effd700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f90780a3d50 0x7f90780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.704+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f90800d0380 con 0x7f9090103f10 2024-04-26T22:49:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.704+0000 7f908effd700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f90780a3d50 0x7f90780a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f908400b5c0 tx=0x7f9084005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:13.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.709+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9080020020 con 0x7f9090103f10 2024-04-26T22:49:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:13 smithi186 ceph-mon[32774]: pgmap v338: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T22:49:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:13 smithi038 ceph-mon[32433]: pgmap v338: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T22:49:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.939+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9090063860 con 0x7f90780a3d50 2024-04-26T22:49:13.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.940+0000 7f908cff9700 1 -- 172.21.15.1:0/1224574496 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9090063860 con 0x7f90780a3d50 2024-04-26T22:49:13.940 INFO:teuthology.orchestra.run.smithi001.stdout:Backend: cephadm 2024-04-26T22:49:13.941 INFO:teuthology.orchestra.run.smithi001.stdout:Available: Yes 2024-04-26T22:49:13.941 INFO:teuthology.orchestra.run.smithi001.stdout:Paused: No 2024-04-26T22:49:13.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.942+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f90780a3d50 msgr2=0x7f90780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:13.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.942+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f90780a3d50 0x7f90780a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f908400b5c0 tx=0x7f9084005fb0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 msgr2=0x7f9090103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:13.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 0x7f9090103870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f908000d980 tx=0x7f908000dc90 comp rx=0 tx=0).stop 2024-04-26T22:49:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 shutdown_connections 2024-04-26T22:49:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f90780a3d50 0x7f90780a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9090109170 0x7f9090102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9090103f10 0x7f9090103870 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 --2- 172.21.15.1:0/1224574496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9090106840 0x7f9090101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 >> 172.21.15.1:0/1224574496 conn(0x7f90900fd940 msgr2=0x7f9090104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 shutdown_connections 2024-04-26T22:49:13.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:13.943+0000 7f9095ea3700 1 -- 172.21.15.1:0/1224574496 wait complete. 2024-04-26T22:49:14.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T22:49:14.973 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:15 smithi186 ceph-mon[32774]: pgmap v339: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T22:49:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:15 smithi186 ceph-mon[32774]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:15 smithi001 ceph-mon[29435]: pgmap v339: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T22:49:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:15 smithi001 ceph-mon[29435]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:15 smithi038 ceph-mon[32433]: pgmap v339: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T22:49:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:15 smithi038 ceph-mon[32433]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 -- 172.21.15.1:0/976464211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70104680 msgr2=0x7f2b70104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 --2- 172.21.15.1:0/976464211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70104680 0x7f2b70104ae0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2b640099c0 tx=0x7f2b64009cd0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 -- 172.21.15.1:0/976464211 shutdown_connections 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 --2- 172.21.15.1:0/976464211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70105020 0x7f2b7010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 --2- 172.21.15.1:0/976464211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b70103480 0x7f2b701038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 --2- 172.21.15.1:0/976464211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70104680 0x7f2b70104ae0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 -- 172.21.15.1:0/976464211 >> 172.21.15.1:0/976464211 conn(0x7f2b700fea20 msgr2=0x7f2b70100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 -- 172.21.15.1:0/976464211 shutdown_connections 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.647+0000 7f2b75008700 1 -- 172.21.15.1:0/976464211 wait complete. 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 Processor -- start 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 -- start start 2024-04-26T22:49:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 0x7f2b700771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b70104680 0x7f2b70075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70105020 0x7f2b70075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7010f010 con 0x7f2b70103480 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7010ee90 con 0x7f2b70104680 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b75008700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7010f190 con 0x7f2b70105020 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b6ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 0x7f2b700771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b6ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 0x7f2b700771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43702/0 (socket says 172.21.15.1:43702) 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b6ed9d700 1 -- 172.21.15.1:0/2547129076 learned_addr learned my addr 172.21.15.1:0/2547129076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b6e59c700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b70104680 0x7f2b70075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.648+0000 7f2b6f59e700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70105020 0x7f2b70075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b6ed9d700 1 -- 172.21.15.1:0/2547129076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70105020 msgr2=0x7f2b70075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b6ed9d700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70105020 0x7f2b70075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b6ed9d700 1 -- 172.21.15.1:0/2547129076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b70104680 msgr2=0x7f2b70075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b6ed9d700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b70104680 0x7f2b70075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b6ed9d700 1 -- 172.21.15.1:0/2547129076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b70076600 con 0x7f2b70103480 2024-04-26T22:49:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b6ed9d700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 0x7f2b700771c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000b500 tx=0x7f2b6000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60015020 con 0x7f2b70103480 2024-04-26T22:49:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b60011ce0 con 0x7f2b70103480 2024-04-26T22:49:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b701a3820 con 0x7f2b70103480 2024-04-26T22:49:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60010c80 con 0x7f2b70103480 2024-04-26T22:49:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.649+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b701a3db0 con 0x7f2b70103480 2024-04-26T22:49:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.650+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b7006a110 con 0x7f2b70103480 2024-04-26T22:49:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.651+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2b60018020 con 0x7f2b70103480 2024-04-26T22:49:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.652+0000 7f2b57fff700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2b580a3cd0 0x7f2b580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.652+0000 7f2b6e59c700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2b580a3cd0 0x7f2b580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.652+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f2b600cfba0 con 0x7f2b70103480 2024-04-26T22:49:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.653+0000 7f2b6e59c700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2b580a3cd0 0x7f2b580a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009990 tx=0x7f2b6400b910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:16.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.657+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2b6001f020 con 0x7f2b70103480 2024-04-26T22:49:16.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.876+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2b70063860 con 0x7f2b580a3cd0 2024-04-26T22:49:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.890+0000 7f2b57fff700 1 -- 172.21.15.1:0/2547129076 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f2b70063860 con 0x7f2b580a3cd0 2024-04-26T22:49:16.890 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T22:49:16.890 INFO:teuthology.orchestra.run.smithi001.stdout:mds.a smithi001 running (104s) 18s ago 104s 19.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 06e44256bb93 2024-04-26T22:49:16.890 INFO:teuthology.orchestra.run.smithi001.stdout:mds.b smithi038 running (101s) 35s ago 101s 16.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 a479f886c446 2024-04-26T22:49:16.890 INFO:teuthology.orchestra.run.smithi001.stdout:mds.c smithi186 running (116s) 35s ago 115s 17.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 2c82dc6dc1c5 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.d smithi001 running (2m) 18s ago 2m 13.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 6662f60e4110 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.e smithi038 running (2m) 35s ago 2m 17.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 a7cd8edaf247 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.f smithi186 running (118s) 35s ago 118s 17.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 c29e21bd6ef0 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.g smithi001 running (107s) 18s ago 106s 18.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 e60acb5583e3 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.h smithi038 running (112s) 35s ago 112s 19.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 da7eb8145131 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.i smithi186 running (110s) 35s ago 109s 25.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f399c051c443 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.j smithi001 running (2m) 18s ago 2m 13.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 06280d7ead9b 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.k smithi038 running (2m) 35s ago 2m 19.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 b4d7e89d0194 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mds.l smithi186 running (2m) 35s ago 2m 19.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 072a0005294f 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.x smithi001 *:9283,8765 running (9m) 18s ago 9m 515M - 19.0.0-3306-gbc612ffb 55dc54019ad8 2b3a40c98d9a 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.y smithi038 *:8443,8765 running (7m) 35s ago 7m 452M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5c25694dee7b 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.z smithi186 *:8443,8765 running (7m) 35s ago 7m 452M - 19.0.0-3306-gbc612ffb 55dc54019ad8 61802ca1077d 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a smithi001 running (9m) 18s ago 9m 51.1M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 f7d0c6f1dd9e 2024-04-26T22:49:16.891 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b smithi038 running (7m) 35s ago 7m 42.8M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 714e571c3bf1 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c smithi186 running (8m) 35s ago 8m 38.7M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 4d2a25e14b36 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (7m) 18s ago 7m 45.0M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 ddfa7eed59e6 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi038 running (6m) 35s ago 6m 46.4M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 b0d461925e44 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi186 running (6m) 35s ago 6m 57.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 1a142d53f7d2 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi001 running (6m) 18s ago 6m 45.9M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 b3c71b7dcdb2 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi038 running (5m) 35s ago 5m 45.7M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 0f375bd15d7b 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi186 running (5m) 35s ago 5m 55.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 7234b1243646 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6 smithi001 running (4m) 18s ago 4m 52.2M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 906ce795d063 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7 smithi038 running (4m) 35s ago 4m 44.2M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 bdae9158c5c3 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8 smithi186 running (3m) 35s ago 3m 59.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 515ac8cccd69 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9 smithi001 running (3m) 18s ago 3m 44.7M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 1b4fcf60fef5 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10 smithi038 running (3m) 35s ago 3m 42.8M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 c79a8992d11b 2024-04-26T22:49:16.892 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11 smithi186 running (2m) 35s ago 2m 50.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 5ebcd9356c6e 2024-04-26T22:49:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2b580a3cd0 msgr2=0x7f2b580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2b580a3cd0 0x7f2b580a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009990 tx=0x7f2b6400b910 comp rx=0 tx=0).stop 2024-04-26T22:49:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 msgr2=0x7f2b700771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 0x7f2b700771c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000b500 tx=0x7f2b6000b810 comp rx=0 tx=0).stop 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 shutdown_connections 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f2b580a3cd0 0x7f2b580a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70105020 0x7f2b70075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b70103480 0x7f2b700771c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 --2- 172.21.15.1:0/2547129076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b70104680 0x7f2b70075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 >> 172.21.15.1:0/2547129076 conn(0x7f2b700fea20 msgr2=0x7f2b70107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 shutdown_connections 2024-04-26T22:49:16.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:16.892+0000 7f2b75008700 1 -- 172.21.15.1:0/2547129076 wait complete. 2024-04-26T22:49:17.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:17 smithi001 ceph-mon[29435]: pgmap v340: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T22:49:17.592 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T22:49:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:17 smithi186 ceph-mon[32774]: pgmap v340: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T22:49:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:17 smithi038 ceph-mon[32433]: pgmap v340: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T22:49:18.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:18 smithi001 ceph-mon[29435]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:18.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:18 smithi038 ceph-mon[32433]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:18.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:18 smithi186 ceph-mon[32774]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 -- 172.21.15.1:0/2718022435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01046a0 msgr2=0x7efef0104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 --2- 172.21.15.1:0/2718022435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01046a0 0x7efef0104b00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efee40099c0 tx=0x7efee4009cd0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 -- 172.21.15.1:0/2718022435 shutdown_connections 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 --2- 172.21.15.1:0/2718022435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef01034a0 0x7efef01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 --2- 172.21.15.1:0/2718022435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef0105040 0x7efef010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 --2- 172.21.15.1:0/2718022435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01046a0 0x7efef0104b00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 -- 172.21.15.1:0/2718022435 >> 172.21.15.1:0/2718022435 conn(0x7efef00fea40 msgr2=0x7efef0100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 -- 172.21.15.1:0/2718022435 shutdown_connections 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 -- 172.21.15.1:0/2718022435 wait complete. 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 Processor -- start 2024-04-26T22:49:19.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.603+0000 7efef6429700 1 -- start start 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef6429700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 0x7efef00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef6429700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01046a0 0x7efef0075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef6429700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef0105040 0x7efef0075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef6429700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efef010ef40 con 0x7efef01034a0 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef6429700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efef010edc0 con 0x7efef01046a0 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef6429700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efef010f0c0 con 0x7efef0105040 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 0x7efef00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 0x7efef00771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43732/0 (socket says 172.21.15.1:43732) 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 -- 172.21.15.1:0/3000945028 learned_addr learned my addr 172.21.15.1:0/3000945028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efef49c6700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef0105040 0x7efef0075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeef7fe700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01046a0 0x7efef0075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 -- 172.21.15.1:0/3000945028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef0105040 msgr2=0x7efef0075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:19.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef0105040 0x7efef0075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 -- 172.21.15.1:0/3000945028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01046a0 msgr2=0x7efef0075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01046a0 0x7efef0075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 -- 172.21.15.1:0/3000945028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef0076600 con 0x7efef01034a0 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeef7fe700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01046a0 0x7efef0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeeffff700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 0x7efef00771c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efee000d980 tx=0x7efee000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.604+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0017070 con 0x7efef01034a0 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.605+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee0009de0 con 0x7efef01034a0 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.605+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef01a3740 con 0x7efef01034a0 2024-04-26T22:49:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.605+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef01a3c00 con 0x7efef01034a0 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.605+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee000f3c0 con 0x7efef01034a0 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.606+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef006a110 con 0x7efef01034a0 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.607+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7efee0009930 con 0x7efef01034a0 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.607+0000 7efeed7fa700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7efed80a3cd0 0x7efed80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.607+0000 7efeef7fe700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7efed80a3cd0 0x7efed80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.607+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7efee00d0100 con 0x7efef01034a0 2024-04-26T22:49:19.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.608+0000 7efeef7fe700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7efed80a3cd0 0x7efed80a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:19.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.612+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efee009c9b0 con 0x7efef01034a0 2024-04-26T22:49:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.829+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7efef0063860 con 0x7efed80a3cd0 2024-04-26T22:49:19.830 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:19 smithi001 ceph-mon[29435]: pgmap v341: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:19.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.838+0000 7efeed7fa700 1 -- 172.21.15.1:0/3000945028 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7efef0063860 con 0x7efed80a3cd0 2024-04-26T22:49:19.838 INFO:teuthology.orchestra.run.smithi001.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T22:49:19.838 INFO:teuthology.orchestra.run.smithi001.stdout:mds.all 12/12 38s ago 2m smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T22:49:19.839 INFO:teuthology.orchestra.run.smithi001.stdout:mgr 3/3 38s ago 7m smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T22:49:19.839 INFO:teuthology.orchestra.run.smithi001.stdout:mon 3/3 38s ago 8m smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c;count:3 2024-04-26T22:49:19.839 INFO:teuthology.orchestra.run.smithi001.stdout:osd 12 38s ago - 2024-04-26T22:49:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7efed80a3cd0 msgr2=0x7efed80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7efed80a3cd0 0x7efed80a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4005fd0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 msgr2=0x7efef00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 0x7efef00771c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efee000d980 tx=0x7efee000dc90 comp rx=0 tx=0).stop 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 shutdown_connections 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7efed80a3cd0 0x7efed80a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef0105040 0x7efef0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef01034a0 0x7efef00771c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 --2- 172.21.15.1:0/3000945028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01046a0 0x7efef0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 >> 172.21.15.1:0/3000945028 conn(0x7efef00fea40 msgr2=0x7efef0107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 shutdown_connections 2024-04-26T22:49:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:19.840+0000 7efef6429700 1 -- 172.21.15.1:0/3000945028 wait complete. 2024-04-26T22:49:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:19 smithi038 ceph-mon[32433]: pgmap v341: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:19 smithi186 ceph-mon[32774]: pgmap v341: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:20.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T22:49:20.861 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:21 smithi001 ceph-mon[29435]: pgmap v342: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:21 smithi001 ceph-mon[29435]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:21 smithi038 ceph-mon[32433]: pgmap v342: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:21 smithi038 ceph-mon[32433]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:21 smithi186 ceph-mon[32774]: pgmap v342: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:21 smithi186 ceph-mon[32774]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 -- 172.21.15.1:0/1769968316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8103f50 msgr2=0x7f4fa8106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/1769968316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8103f50 0x7f4fa8106340 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009990 tx=0x7f4f98009ca0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 -- 172.21.15.1:0/1769968316 shutdown_connections 2024-04-26T22:49:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/1769968316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa81091b0 0x7f4fa810b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/1769968316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8103f50 0x7f4fa8106340 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/1769968316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa8106880 0x7f4fa8108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 -- 172.21.15.1:0/1769968316 >> 172.21.15.1:0/1769968316 conn(0x7f4fa80fd980 msgr2=0x7f4fa80ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 -- 172.21.15.1:0/1769968316 shutdown_connections 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.619+0000 7f4fad1ac700 1 -- 172.21.15.1:0/1769968316 wait complete. 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 Processor -- start 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 -- start start 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8103f50 0x7f4fa81038b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 0x7f4fa8101f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa81091b0 0x7f4fa8102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa810efc0 con 0x7f4fa8106880 2024-04-26T22:49:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa810ee40 con 0x7f4fa8103f50 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4fad1ac700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa810f140 con 0x7f4fa81091b0 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4f9ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 0x7f4fa8101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4f9ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 0x7f4fa8101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57804/0 (socket says 172.21.15.1:57804) 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.620+0000 7f4f9ffff700 1 -- 172.21.15.1:0/2903613839 learned_addr learned my addr 172.21.15.1:0/2903613839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fa6d9d700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8103f50 0x7f4fa81038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fa759e700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa81091b0 0x7f4fa8102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4f9ffff700 1 -- 172.21.15.1:0/2903613839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa81091b0 msgr2=0x7f4fa8102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4f9ffff700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa81091b0 0x7f4fa8102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4f9ffff700 1 -- 172.21.15.1:0/2903613839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8103f50 msgr2=0x7f4fa81038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4f9ffff700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8103f50 0x7f4fa81038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4f9ffff700 1 -- 172.21.15.1:0/2903613839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa8102cf0 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fa759e700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa81091b0 0x7f4fa8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4f9ffff700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 0x7f4fa8101f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b530 tx=0x7f4f9000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90015020 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa81a37f0 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f90011ce0 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90010e40 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.621+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa81a3db0 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.622+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa806a110 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.623+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4f90018020 con 0x7f4fa8106880 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.624+0000 7f4fa4d99700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f940a3cd0 0x7f4f940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.624+0000 7f4fa6d9d700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f940a3cd0 0x7f4f940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.624+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f4f900d0220 con 0x7f4fa8106880 2024-04-26T22:49:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.624+0000 7f4fa6d9d700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f940a3cd0 0x7f4f940a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800b5c0 tx=0x7f4f98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.628+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4f9001f020 con 0x7f4fa8106880 2024-04-26T22:49:22.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.841+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f4fa8063860 con 0x7f4f940a3cd0 2024-04-26T22:49:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.843+0000 7f4fa4d99700 1 -- 172.21.15.1:0/2903613839 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f4fa8063860 con 0x7f4f940a3cd0 2024-04-26T22:49:22.843 INFO:teuthology.orchestra.run.smithi001.stdout:HOST ADDR LABELS STATUS 2024-04-26T22:49:22.843 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 172.21.15.1 2024-04-26T22:49:22.843 INFO:teuthology.orchestra.run.smithi001.stdout:smithi038 172.21.15.38 2024-04-26T22:49:22.843 INFO:teuthology.orchestra.run.smithi001.stdout:smithi186 172.21.15.186 2024-04-26T22:49:22.843 INFO:teuthology.orchestra.run.smithi001.stdout:3 hosts in cluster 2024-04-26T22:49:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f940a3cd0 msgr2=0x7f4f940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f940a3cd0 0x7f4f940a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800b5c0 tx=0x7f4f98005fb0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 msgr2=0x7f4fa8101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 0x7f4fa8101f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b530 tx=0x7f4f9000b840 comp rx=0 tx=0).stop 2024-04-26T22:49:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 shutdown_connections 2024-04-26T22:49:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f940a3cd0 0x7f4f940a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa81091b0 0x7f4fa8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8103f50 0x7f4fa81038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 --2- 172.21.15.1:0/2903613839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa8106880 0x7f4fa8101f00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 >> 172.21.15.1:0/2903613839 conn(0x7f4fa80fd980 msgr2=0x7f4fa81048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 shutdown_connections 2024-04-26T22:49:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:22.845+0000 7f4fad1ac700 1 -- 172.21.15.1:0/2903613839 wait complete. 2024-04-26T22:49:23.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T22:49:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:23 smithi001 ceph-mon[29435]: pgmap v343: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:23 smithi038 ceph-mon[32433]: pgmap v343: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:23.928 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:23 smithi186 ceph-mon[32774]: pgmap v343: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:24.758 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:24 smithi001 ceph-mon[29435]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:24.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:24 smithi038 ceph-mon[32433]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:24.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:24 smithi186 ceph-mon[32774]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:25.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.611+0000 7f8e8578d700 1 -- 172.21.15.1:0/3330013536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 msgr2=0x7f8e801038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:25.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.611+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3330013536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801038c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8e700099c0 tx=0x7f8e70009cd0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 -- 172.21.15.1:0/3330013536 shutdown_connections 2024-04-26T22:49:25.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3330013536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e80105040 0x7f8e8010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3330013536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801038c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3330013536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e801046a0 0x7f8e80104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 -- 172.21.15.1:0/3330013536 >> 172.21.15.1:0/3330013536 conn(0x7f8e800fea40 msgr2=0x7f8e80100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:25.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 -- 172.21.15.1:0/3330013536 shutdown_connections 2024-04-26T22:49:25.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 -- 172.21.15.1:0/3330013536 wait complete. 2024-04-26T22:49:25.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 Processor -- start 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.612+0000 7f8e8578d700 1 -- start start 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e801046a0 0x7f8e80199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e80105040 0x7f8e801a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8010ef30 con 0x7f8e801034a0 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8010edb0 con 0x7f8e801046a0 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8010f0b0 con 0x7f8e80105040 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57824/0 (socket says 172.21.15.1:57824) 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 -- 172.21.15.1:0/3256199206 learned_addr learned my addr 172.21.15.1:0/3256199206 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:25.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7e7fc700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e801046a0 0x7f8e80199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:25.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7f7fe700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e80105040 0x7f8e801a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:25.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 -- 172.21.15.1:0/3256199206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e80105040 msgr2=0x7f8e801a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:25.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e80105040 0x7f8e801a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 -- 172.21.15.1:0/3256199206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e801046a0 msgr2=0x7f8e80199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e801046a0 0x7f8e80199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 -- 172.21.15.1:0/3256199206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e801a1230 con 0x7f8e801034a0 2024-04-26T22:49:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e7effd700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801995d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009f50 tx=0x7f8e700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70027070 con 0x7f8e801034a0 2024-04-26T22:49:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e801a14c0 con 0x7f8e801034a0 2024-04-26T22:49:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e70004020 con 0x7f8e801034a0 2024-04-26T22:49:25.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.613+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70022ca0 con 0x7f8e801034a0 2024-04-26T22:49:25.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.614+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e801a19f0 con 0x7f8e801034a0 2024-04-26T22:49:25.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.614+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e8006a110 con 0x7f8e801034a0 2024-04-26T22:49:25.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.616+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8e7002a070 con 0x7f8e801034a0 2024-04-26T22:49:25.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.616+0000 7f8e67fff700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8e680a3cd0 0x7f8e680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:25.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.616+0000 7f8e7e7fc700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8e680a3cd0 0x7f8e680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:25.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.617+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f8e700d18a0 con 0x7f8e801034a0 2024-04-26T22:49:25.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.617+0000 7f8e7e7fc700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8e680a3cd0 0x7f8e680a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8e74005950 tx=0x7f8e7400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:25.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.620+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e70031070 con 0x7f8e801034a0 2024-04-26T22:49:25.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.839+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f8e80063860 con 0x7f8e680a3cd0 2024-04-26T22:49:25.840 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:25 smithi001 ceph-mon[29435]: pgmap v344: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:25.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.841+0000 7f8e67fff700 1 -- 172.21.15.1:0/3256199206 <== mgr.14152 v2:172.21.15.1:6800/4125558078 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f8e80063860 con 0x7f8e680a3cd0 2024-04-26T22:49:25.841 INFO:teuthology.orchestra.run.smithi001.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T22:49:25.841 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513000D480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-26T22:49:25.841 INFO:teuthology.orchestra.run.smithi001.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-26T22:49:25.841 INFO:teuthology.orchestra.run.smithi001.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8e680a3cd0 msgr2=0x7f8e680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8e680a3cd0 0x7f8e680a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8e74005950 tx=0x7f8e7400b410 comp rx=0 tx=0).stop 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 msgr2=0x7f8e801995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801995d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009f50 tx=0x7f8e700047b0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 shutdown_connections 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f8e680a3cd0 0x7f8e680a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e80105040 0x7f8e801a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e801034a0 0x7f8e801995d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 --2- 172.21.15.1:0/3256199206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e801046a0 0x7f8e80199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 >> 172.21.15.1:0/3256199206 conn(0x7f8e800fea40 msgr2=0x7f8e80107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 shutdown_connections 2024-04-26T22:49:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:25.843+0000 7f8e8578d700 1 -- 172.21.15.1:0/3256199206 wait complete. 2024-04-26T22:49:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:25 smithi038 ceph-mon[32433]: pgmap v344: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:25 smithi186 ceph-mon[32774]: pgmap v344: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:26.687 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T22:49:26.696 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T22:49:26.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T22:49:26.952 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:27 smithi001 ceph-mon[29435]: pgmap v345: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:27 smithi001 ceph-mon[29435]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:27 smithi038 ceph-mon[32433]: pgmap v345: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:27 smithi038 ceph-mon[32433]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:27 smithi186 ceph-mon[32774]: pgmap v345: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:27 smithi186 ceph-mon[32774]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 -- 172.21.15.1:0/3295686967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 msgr2=0x7f13ac108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 --2- 172.21.15.1:0/3295686967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac108c50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f139c009990 tx=0x7f139c009ca0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 -- 172.21.15.1:0/3295686967 shutdown_connections 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 --2- 172.21.15.1:0/3295686967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac109190 0x7f13ac10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 --2- 172.21.15.1:0/3295686967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13ac103f30 0x7f13ac106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 --2- 172.21.15.1:0/3295686967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac108c50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 -- 172.21.15.1:0/3295686967 >> 172.21.15.1:0/3295686967 conn(0x7f13ac0fd940 msgr2=0x7f13ac0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 -- 172.21.15.1:0/3295686967 shutdown_connections 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.705+0000 7f13b22a8700 1 -- 172.21.15.1:0/3295686967 wait complete. 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 Processor -- start 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 -- start start 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac103f30 0x7f13ac103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13ac109190 0x7f13ac102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10eff0 con 0x7f13ac106860 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10ee70 con 0x7f13ac109190 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b22a8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10f170 con 0x7f13ac103f30 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac101ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57842/0 (socket says 172.21.15.1:57842) 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13ab7fe700 1 -- 172.21.15.1:0/1983075089 learned_addr learned my addr 172.21.15.1:0/1983075089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13b0845700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13ac109190 0x7f13ac102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13abfff700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac103f30 0x7f13ac103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.706+0000 7f13ab7fe700 1 -- 172.21.15.1:0/1983075089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac103f30 msgr2=0x7f13ac103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13ab7fe700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac103f30 0x7f13ac103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13ab7fe700 1 -- 172.21.15.1:0/1983075089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13ac109190 msgr2=0x7f13ac102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13ab7fe700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13ac109190 0x7f13ac102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13ab7fe700 1 -- 172.21.15.1:0/1983075089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac102cd0 con 0x7f13ac106860 2024-04-26T22:49:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13abfff700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac103f30 0x7f13ac103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13ab7fe700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac101ee0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f139c0036f0 tx=0x7f139c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c027070 con 0x7f13ac106860 2024-04-26T22:49:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac1a3730 con 0x7f13ac106860 2024-04-26T22:49:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c004100 con 0x7f13ac106860 2024-04-26T22:49:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c022e60 con 0x7f13ac106860 2024-04-26T22:49:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.707+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac1a3bb0 con 0x7f13ac106860 2024-04-26T22:49:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.708+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ac06a110 con 0x7f13ac106860 2024-04-26T22:49:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.709+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f139c02a070 con 0x7f13ac106860 2024-04-26T22:49:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.710+0000 7f13a97fa700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13980a3cd0 0x7f13980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.710+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f139c0d19c0 con 0x7f13ac106860 2024-04-26T22:49:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.710+0000 7f13abfff700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13980a3cd0 0x7f13980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.710+0000 7f13abfff700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13980a3cd0 0x7f13980a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1394007900 tx=0x7f1394008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.713+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f139c013070 con 0x7f13ac106860 2024-04-26T22:49:28.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.967+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f13ac02e560 con 0x7f13ac106860 2024-04-26T22:49:28.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.968+0000 7f13a97fa700 1 -- 172.21.15.1:0/1983075089 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3080 (secure 0 0 0) 0x7f139c09e300 con 0x7f13ac106860 2024-04-26T22:49:28.968 INFO:teuthology.orchestra.run.smithi001.stdout:e25 2024-04-26T22:49:28.968 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T22:49:28.968 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T22:49:28.968 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 25 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-04-26T22:47:41.130042+0000 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-04-26T22:47:47.801060+0000 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-04-26T22:49:28.969 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 300 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 5 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1,2,3,4 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=24514,1=24487,2=24475,3=24473,4=14631} 2024-04-26T22:49:28.970 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data disabled 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 0 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:qdb_cluster leader: 24514 members: 24475,24514,24487,24473,14631 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.i{0:24514} state up:active seq 5 addr [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.e{0:24440} state up:standby-replay seq 1 addr [v2:172.21.15.38:6834/1362256204,v1:172.21.15.38:6835/1362256204] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c{1:24487} state up:active seq 7 addr [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.l{1:24427} state up:standby-replay seq 1 addr [v2:172.21.15.186:6832/4165972801,v1:172.21.15.186:6833/4165972801] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.971 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.f{2:24475} state up:active seq 8 addr [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.k{2:24413} state up:standby-replay seq 1 addr [v2:172.21.15.38:6832/3943944853,v1:172.21.15.38:6833/3943944853] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.h{3:24473} state up:active seq 7 addr [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b{3:14643} state up:standby-replay seq 1 addr [v2:172.21.15.38:6838/2490292790,v1:172.21.15.38:6839/2490292790] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a{4:14631} state up:active seq 5 addr [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.g{4:14619} state up:standby-replay seq 1 addr [v2:172.21.15.1:6838/1512535471,v1:172.21.15.1:6839/1512535471] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.1:6834/2653371918,v1:172.21.15.1:6835/2653371918] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.1:6836/1791309239,v1:172.21.15.1:6837/1791309239] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T22:49:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13980a3cd0 msgr2=0x7f13980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13980a3cd0 0x7f13980a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1394007900 tx=0x7f1394008040 comp rx=0 tx=0).stop 2024-04-26T22:49:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 msgr2=0x7f13ac101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac101ee0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f139c0036f0 tx=0x7f139c005f10 comp rx=0 tx=0).stop 2024-04-26T22:49:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 shutdown_connections 2024-04-26T22:49:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13980a3cd0 0x7f13980a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13ac103f30 0x7f13ac103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13ac109190 0x7f13ac102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 --2- 172.21.15.1:0/1983075089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac106860 0x7f13ac101ee0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 >> 172.21.15.1:0/1983075089 conn(0x7f13ac0fd940 msgr2=0x7f13ac104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 shutdown_connections 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:28.970+0000 7f13b22a8700 1 -- 172.21.15.1:0/1983075089 wait complete. 2024-04-26T22:49:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:49:29.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:29 smithi001 ceph-mon[29435]: pgmap v346: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:29.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:29 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1983075089' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T22:49:29.787 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:29 smithi186 ceph-mon[32774]: pgmap v346: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:29.787 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:29 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1983075089' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T22:49:29.804 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T22:49:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:29 smithi038 ceph-mon[32433]: pgmap v346: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:29 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1983075089' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T22:49:30.054 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.674+0000 7f8314100700 1 -- 172.21.15.1:0/3181556690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 msgr2=0x7f830c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.674+0000 7f8314100700 1 --2- 172.21.15.1:0/3181556690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c10a7f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f830400b0d0 tx=0x7f830400b3e0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- 172.21.15.1:0/3181556690 shutdown_connections 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 --2- 172.21.15.1:0/3181556690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1023a0 0x7f830c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 --2- 172.21.15.1:0/3181556690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c10a7f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 --2- 172.21.15.1:0/3181556690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1035a0 0x7f830c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- 172.21.15.1:0/3181556690 >> 172.21.15.1:0/3181556690 conn(0x7f830c0fd940 msgr2=0x7f830c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- 172.21.15.1:0/3181556690 shutdown_connections 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- 172.21.15.1:0/3181556690 wait complete. 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 Processor -- start 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- start start 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1023a0 0x7f830c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1035a0 0x7f830c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c1a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f830c10df30 con 0x7f830c103f40 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.675+0000 7f8314100700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f830c10ddb0 con 0x7f830c1023a0 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8314100700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f830c10e0b0 con 0x7f830c1035a0 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c1a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c1a2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57866/0 (socket says 172.21.15.1:57866) 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 -- 172.21.15.1:0/1684423620 learned_addr learned my addr 172.21.15.1:0/1684423620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8311e9c700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1023a0 0x7f830c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831169b700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1035a0 0x7f830c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 -- 172.21.15.1:0/1684423620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1035a0 msgr2=0x7f830c19be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1035a0 0x7f830c19be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 -- 172.21.15.1:0/1684423620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1023a0 msgr2=0x7f830c19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1023a0 0x7f830c19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 -- 172.21.15.1:0/1684423620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830c1a3530 con 0x7f830c103f40 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831169b700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1035a0 0x7f830c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8311e9c700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1023a0 0x7f830c19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f831269d700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c1a2e90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f830400d010 tx=0x7f8304004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830401f030 con 0x7f830c103f40 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8304007be0 con 0x7f830c103f40 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830c1a37c0 con 0x7f830c103f40 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.676+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304004810 con 0x7f830c103f40 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.677+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830c1a3c90 con 0x7f830c103f40 2024-04-26T22:49:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.677+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830c06a110 con 0x7f830c103f40 2024-04-26T22:49:31.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.679+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8304003a00 con 0x7f830c103f40 2024-04-26T22:49:31.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.680+0000 7f8302ffd700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82f80a3cd0 0x7f82f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:31.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.680+0000 7f8311e9c700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82f80a3cd0 0x7f82f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.680+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f830400f020 con 0x7f830c103f40 2024-04-26T22:49:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.680+0000 7f8311e9c700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82f80a3cd0 0x7f82f80a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f830c103400 tx=0x7f8308008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.683+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8304029020 con 0x7f830c103f40 2024-04-26T22:49:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:31 smithi001 ceph-mon[29435]: pgmap v347: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:31 smithi038 ceph-mon[32433]: pgmap v347: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:31.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.891+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f830c1a3f40 con 0x7f830c103f40 2024-04-26T22:49:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.892+0000 7f8302ffd700 1 -- 172.21.15.1:0/1684423620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5190 (secure 0 0 0) 0x7f830c1a3f40 con 0x7f830c103f40 2024-04-26T22:49:31.892 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 87 2024-04-26T22:49:31.892 INFO:teuthology.orchestra.run.smithi001.stdout:fsid bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-04-26T22:39:26.836716+0000 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-04-26T22:48:30.916101+0000 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:crush_version 26 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:full_ratio 0.95 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:backfillfull_ratio 0.9 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:nearfull_ratio 0.85 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:require_min_compat_client luminous 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:min_compat_client luminous 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:require_osd_release squid 2024-04-26T22:49:31.893 INFO:teuthology.orchestra.run.smithi001.stdout:stretch_mode_enabled false 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:max_osd 12 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6802/1864154398,v1:172.21.15.1:6803/1864154398] [v2:172.21.15.1:6804/1864154398,v1:172.21.15.1:6805/1864154398] exists,up 5213229d-410b-4bd5-9fb0-fb58a13e3322 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6800/156758275,v1:172.21.15.38:6801/156758275] [v2:172.21.15.38:6802/156758275,v1:172.21.15.38:6803/156758275] exists,up efa8c529-7d81-4be2-90ee-2ce0cc461c84 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/2937417141,v1:172.21.15.186:6801/2937417141] [v2:172.21.15.186:6802/2937417141,v1:172.21.15.186:6803/2937417141] exists,up 7d043997-5257-4a78-99a6-4e3483360b0a 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6810/10840892,v1:172.21.15.1:6811/10840892] [v2:172.21.15.1:6812/10840892,v1:172.21.15.1:6813/10840892] exists,up ed1bef0e-ba55-4c4a-9423-0024c218b1dc 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6808/414739057,v1:172.21.15.38:6809/414739057] [v2:172.21.15.38:6810/414739057,v1:172.21.15.38:6811/414739057] exists,up 98421e8d-87a9-4a37-9415-d806e7d31651 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/1470594715,v1:172.21.15.186:6809/1470594715] [v2:172.21.15.186:6810/1470594715,v1:172.21.15.186:6811/1470594715] exists,up f6b59471-baf0-4b9b-8756-43142ee043fe 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6818/244390713,v1:172.21.15.1:6819/244390713] [v2:172.21.15.1:6820/244390713,v1:172.21.15.1:6821/244390713] exists,up cc08c46f-8ce2-448c-a9fb-74b7559b0d19 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6816/519750991,v1:172.21.15.38:6817/519750991] [v2:172.21.15.38:6818/519750991,v1:172.21.15.38:6819/519750991] exists,up 579e3ab3-4803-4c10-835f-266ce4f811ec 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/955115503,v1:172.21.15.186:6817/955115503] [v2:172.21.15.186:6818/955115503,v1:172.21.15.186:6819/955115503] exists,up 0e754df8-e30f-40bf-b754-351c414e8767 2024-04-26T22:49:31.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6826/2190648434,v1:172.21.15.1:6827/2190648434] [v2:172.21.15.1:6828/2190648434,v1:172.21.15.1:6829/2190648434] exists,up f781b994-298e-413f-9391-b9de190549f8 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6824/1475724574,v1:172.21.15.38:6825/1475724574] [v2:172.21.15.38:6826/1475724574,v1:172.21.15.38:6827/1475724574] exists,up 9cbc770f-5853-41c8-9be5-c5b2dcec7fcc 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/63751610,v1:172.21.15.186:6825/63751610] [v2:172.21.15.186:6826/63751610,v1:172.21.15.186:6827/63751610] exists,up 7e7a126e-d734-4b97-aa0e-6a547cb69575 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.8 [2,11] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.f [8,11] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.0 [5,9] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.13 [4,9] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.1d [4,8] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.31 [4,6] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.38 [3,9] 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/2466076666 expires 2024-04-27T22:40:29.666140+0000 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6801/3267707627 expires 2024-04-27T22:40:29.666140+0000 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6800/3267707627 expires 2024-04-27T22:40:29.666140+0000 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/2458055359 expires 2024-04-27T22:40:29.666140+0000 2024-04-26T22:49:31.895 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/717752603 expires 2024-04-27T22:40:02.607868+0000 2024-04-26T22:49:31.896 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/4224349841 expires 2024-04-27T22:40:02.607868+0000 2024-04-26T22:49:31.896 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6800/722669381 expires 2024-04-27T22:40:02.607868+0000 2024-04-26T22:49:31.896 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/3044795811 expires 2024-04-27T22:40:02.607868+0000 2024-04-26T22:49:31.896 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6801/722669381 expires 2024-04-27T22:40:02.607868+0000 2024-04-26T22:49:31.896 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/3240847897 expires 2024-04-27T22:40:29.666140+0000 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82f80a3cd0 msgr2=0x7f82f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82f80a3cd0 0x7f82f80a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f830c103400 tx=0x7f8308008040 comp rx=0 tx=0).stop 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 msgr2=0x7f830c1a2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c1a2e90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f830400d010 tx=0x7f8304004ef0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 shutdown_connections 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f82f80a3cd0 0x7f82f80a6190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f830c103f40 0x7f830c1a2e90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f830c1023a0 0x7f830c19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 --2- 172.21.15.1:0/1684423620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f830c1035a0 0x7f830c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 >> 172.21.15.1:0/1684423620 conn(0x7f830c0fd940 msgr2=0x7f830c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 shutdown_connections 2024-04-26T22:49:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:31.894+0000 7f8314100700 1 -- 172.21.15.1:0/1684423620 wait complete. 2024-04-26T22:49:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:31 smithi186 ceph-mon[32774]: pgmap v347: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:32.727 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T22:49:32.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:49:32.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:32 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1684423620' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 -- 172.21.15.1:0/2294051258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8105d40 msgr2=0x7f46d8108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 --2- 172.21.15.1:0/2294051258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8105d40 0x7f46d8108130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009900 tx=0x7f46c802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 -- 172.21.15.1:0/2294051258 shutdown_connections 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 --2- 172.21.15.1:0/2294051258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46d8108670 0x7f46d810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 --2- 172.21.15.1:0/2294051258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8105d40 0x7f46d8108130 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 --2- 172.21.15.1:0/2294051258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46d8103410 0x7f46d8105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 -- 172.21.15.1:0/2294051258 >> 172.21.15.1:0/2294051258 conn(0x7f46d80fce00 msgr2=0x7f46d80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 -- 172.21.15.1:0/2294051258 shutdown_connections 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 -- 172.21.15.1:0/2294051258 wait complete. 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 Processor -- start 2024-04-26T22:49:32.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.846+0000 7f46df389640 1 -- start start 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46d8103410 0x7f46d8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46d8105d40 0x7f46d8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 0x7f46d81a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f46d810e4b0 con 0x7f46d8108670 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f46d810e330 con 0x7f46d8105d40 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46d810e630 con 0x7f46d8103410 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 0x7f46d81a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 0x7f46d81a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42772/0 (socket says 172.21.15.1:42772) 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 -- 172.21.15.1:0/3578127226 learned_addr learned my addr 172.21.15.1:0/3578127226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dc924640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46d8105d40 0x7f46d8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd125640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46d8103410 0x7f46d8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:32.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 -- 172.21.15.1:0/3578127226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46d8103410 msgr2=0x7f46d8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46d8103410 0x7f46d8198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 -- 172.21.15.1:0/3578127226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46d8105d40 msgr2=0x7f46d8199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46d8105d40 0x7f46d8199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 -- 172.21.15.1:0/3578127226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d81a0800 con 0x7f46d8108670 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46dd926640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 0x7f46d81a0190 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009fd0 tx=0x7f46d000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0017070 con 0x7f46d8108670 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d81a0af0 con 0x7f46d8108670 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d0003e20 con 0x7f46d8108670 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0010460 con 0x7f46d8108670 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.847+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d81a1010 con 0x7f46d8108670 2024-04-26T22:49:32.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.848+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d80fe8f0 con 0x7f46d8108670 2024-04-26T22:49:32.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:32 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1684423620' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T22:49:32.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.849+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f46d0003970 con 0x7f46d8108670 2024-04-26T22:49:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.850+0000 7f46ce7fc640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f46c40a3c40 0x7f46c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.850+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f46d00d0310 con 0x7f46d8108670 2024-04-26T22:49:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.850+0000 7f46dd125640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f46c40a3c40 0x7f46c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.851+0000 7f46dd125640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f46c40a3c40 0x7f46c40a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f46d4005fd0 tx=0x7f46d4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:32.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:32.855+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f46d009cbc0 con 0x7f46d8108670 2024-04-26T22:49:32.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:32 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1684423620' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T22:49:33.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.041+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46d802dc80 con 0x7f46d8108670 2024-04-26T22:49:33.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.043+0000 7f46ce7fc640 1 -- 172.21.15.1:0/3578127226 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f46d009c4c0 con 0x7f46d8108670 2024-04-26T22:49:33.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:33.044 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:49:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f46c40a3c40 msgr2=0x7f46c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f46c40a3c40 0x7f46c40a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f46d4005fd0 tx=0x7f46d4005850 comp rx=0 tx=0).stop 2024-04-26T22:49:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 msgr2=0x7f46d81a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 0x7f46d81a0190 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009fd0 tx=0x7f46d000b260 comp rx=0 tx=0).stop 2024-04-26T22:49:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 shutdown_connections 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f46c40a3c40 0x7f46c40a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d8108670 0x7f46d81a0190 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46d8105d40 0x7f46d8199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 --2- 172.21.15.1:0/3578127226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46d8103410 0x7f46d8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 >> 172.21.15.1:0/3578127226 conn(0x7f46d80fce00 msgr2=0x7f46d8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 shutdown_connections 2024-04-26T22:49:33.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.045+0000 7f46df389640 1 -- 172.21.15.1:0/3578127226 wait complete. 2024-04-26T22:49:33.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T22:47:39.631594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T22:47:40.081969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:49:33.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T22:49:33.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- 172.21.15.1:0/975622695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0108650 msgr2=0x7fc4f010aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- 172.21.15.1:0/975622695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0108650 0x7fc4f010aaa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8009ca0 tx=0x7fc4e802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- 172.21.15.1:0/975622695 shutdown_connections 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- 172.21.15.1:0/975622695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0108650 0x7fc4f010aaa0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- 172.21.15.1:0/975622695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f0105d20 0x7fc4f0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- 172.21.15.1:0/975622695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4f01033f0 0x7fc4f01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- 172.21.15.1:0/975622695 >> 172.21.15.1:0/975622695 conn(0x7fc4f00fce00 msgr2=0x7fc4f00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- 172.21.15.1:0/975622695 shutdown_connections 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- 172.21.15.1:0/975622695 wait complete. 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 Processor -- start 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- start start 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4f01033f0 0x7fc4f0198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 0x7fc4f01990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f0108650 0x7fc4f01a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f010e4c0 con 0x7fc4f0105d20 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f010e340 con 0x7fc4f0108650 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.162+0000 7fc4f7178640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f010e640 con 0x7fc4f01033f0 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 0x7fc4f01990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 0x7fc4f01990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42796/0 (socket says 172.21.15.1:42796) 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 -- 172.21.15.1:0/4285977365 learned_addr learned my addr 172.21.15.1:0/4285977365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 -- 172.21.15.1:0/4285977365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4f01033f0 msgr2=0x7fc4f0198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4f4f14640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4f01033f0 0x7fc4f0198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4f01033f0 0x7fc4f0198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4f5715640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f0108650 0x7fc4f01a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 -- 172.21.15.1:0/4285977365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f0108650 msgr2=0x7fc4f01a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f0108650 0x7fc4f01a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 -- 172.21.15.1:0/4285977365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f01a07b0 con 0x7fc4f0105d20 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4effff640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 0x7fc4f01990d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00d770 tx=0x7fc4dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc017070 con 0x7fc4f0105d20 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4dc009c30 con 0x7fc4f0105d20 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f01a0aa0 con 0x7fc4f0105d20 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc00f620 con 0x7fc4f0105d20 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.163+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f01a1040 con 0x7fc4f0105d20 2024-04-26T22:49:33.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.164+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f00fe8f0 con 0x7fc4f0105d20 2024-04-26T22:49:33.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.165+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc4dc01a070 con 0x7fc4f0105d20 2024-04-26T22:49:33.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.166+0000 7fc4edffb640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc4e00a3cc0 0x7fc4e00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.166+0000 7fc4f4f14640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc4e00a3cc0 0x7fc4e00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.166+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc4dc0d00d0 con 0x7fc4f0105d20 2024-04-26T22:49:33.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.167+0000 7fc4f4f14640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc4e00a3cc0 0x7fc4e00a6180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40058c0 tx=0x7fc4e4005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:33.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.171+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc4dc009de0 con 0x7fc4f0105d20 2024-04-26T22:49:33.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.355+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc4f002dd10 con 0x7fc4f0105d20 2024-04-26T22:49:33.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.356+0000 7fc4edffb640 1 -- 172.21.15.1:0/4285977365 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fc4dc09c980 con 0x7fc4f0105d20 2024-04-26T22:49:33.357 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:49:33.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc4e00a3cc0 msgr2=0x7fc4e00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc4e00a3cc0 0x7fc4e00a6180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40058c0 tx=0x7fc4e4005c60 comp rx=0 tx=0).stop 2024-04-26T22:49:33.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 msgr2=0x7fc4f01990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 0x7fc4f01990d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00d770 tx=0x7fc4dc00dc30 comp rx=0 tx=0).stop 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 shutdown_connections 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc4e00a3cc0 0x7fc4e00a6180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f0108650 0x7fc4f01a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f0105d20 0x7fc4f01990d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 --2- 172.21.15.1:0/4285977365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4f01033f0 0x7fc4f0198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 >> 172.21.15.1:0/4285977365 conn(0x7fc4f00fce00 msgr2=0x7fc4f0103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 shutdown_connections 2024-04-26T22:49:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.358+0000 7fc4f7178640 1 -- 172.21.15.1:0/4285977365 wait complete. 2024-04-26T22:49:33.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2467254581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0108660 msgr2=0x7f30d010aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2467254581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0108660 0x7f30d010aab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009ca0 tx=0x7f30c802a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2467254581 shutdown_connections 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2467254581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0108660 0x7f30d010aab0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2467254581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d0105d30 0x7f30d0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2467254581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d0103400 0x7f30d01057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2467254581 >> 172.21.15.1:0/2467254581 conn(0x7f30d00fce10 msgr2=0x7f30d00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2467254581 shutdown_connections 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2467254581 wait complete. 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 Processor -- start 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.474+0000 7f30d6ea9640 1 -- start start 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 0x7f30d0198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d0105d30 0x7f30d01990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d0108660 0x7f30d01a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f30d010e4b0 con 0x7f30d0103400 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f30d010e330 con 0x7f30d0108660 2024-04-26T22:49:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f30d010e630 con 0x7f30d0105d30 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 0x7f30d0198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 0x7f30d0198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42808/0 (socket says 172.21.15.1:42808) 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 -- 172.21.15.1:0/2796062723 learned_addr learned my addr 172.21.15.1:0/2796062723 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30cffff640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d0105d30 0x7f30d01990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 -- 172.21.15.1:0/2796062723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d0105d30 msgr2=0x7f30d01990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d0105d30 0x7f30d01990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 -- 172.21.15.1:0/2796062723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d0108660 msgr2=0x7f30d01a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d0108660 0x7f30d01a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 -- 172.21.15.1:0/2796062723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d01a07a0 con 0x7f30d0103400 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d4c45640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 0x7f30d0198b80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f30c400d950 tx=0x7f30c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4017070 con 0x7f30d0103400 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30c4009c30 con 0x7f30d0103400 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c400f620 con 0x7f30d0103400 2024-04-26T22:49:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.475+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d01a0a90 con 0x7f30d0103400 2024-04-26T22:49:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.476+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d01a1030 con 0x7f30d0103400 2024-04-26T22:49:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.476+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d00fe900 con 0x7f30d0103400 2024-04-26T22:49:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.477+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f30c4010460 con 0x7f30d0103400 2024-04-26T22:49:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.478+0000 7f30cdffb640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30c00a3cc0 0x7f30c00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.478+0000 7f30cffff640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30c00a3cc0 0x7f30c00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.478+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f30c40d01c0 con 0x7f30d0103400 2024-04-26T22:49:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.479+0000 7f30cffff640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30c00a3cc0 0x7f30c00a6180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f30d019a090 tx=0x7f30bc00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:33.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.483+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f30c409ca70 con 0x7f30d0103400 2024-04-26T22:49:33.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.670+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f30d002dd50 con 0x7f30d0103400 2024-04-26T22:49:33.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.670+0000 7f30cdffb640 1 -- 172.21.15.1:0/2796062723 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f30c4009de0 con 0x7f30d0103400 2024-04-26T22:49:33.671 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30c00a3cc0 msgr2=0x7f30c00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30c00a3cc0 0x7f30c00a6180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f30d019a090 tx=0x7f30bc00a4c0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 msgr2=0x7f30d0198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 0x7f30d0198b80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f30c400d950 tx=0x7f30c400de10 comp rx=0 tx=0).stop 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 shutdown_connections 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f30c00a3cc0 0x7f30c00a6180 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d0105d30 0x7f30d01990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d0108660 0x7f30d01a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 --2- 172.21.15.1:0/2796062723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d0103400 0x7f30d0198b80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 >> 172.21.15.1:0/2796062723 conn(0x7f30d00fce10 msgr2=0x7f30d0103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 shutdown_connections 2024-04-26T22:49:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.672+0000 7f30d6ea9640 1 -- 172.21.15.1:0/2796062723 wait complete. 2024-04-26T22:49:33.683 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:33 smithi001 ceph-mon[29435]: pgmap v348: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:33.683 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:33 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3578127226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:33.683 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:33 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/4285977365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:33.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T22:49:33.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/2570893370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98104540 msgr2=0x7f4a9810add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/2570893370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98104540 0x7f4a9810add0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009ca0 tx=0x7f4a9002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/2570893370 shutdown_connections 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/2570893370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98104540 0x7f4a9810add0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/2570893370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a98103b80 0x7f4a98104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/2570893370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a98102990 0x7f4a98102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/2570893370 >> 172.21.15.1:0/2570893370 conn(0x7f4a980fdf00 msgr2=0x7f4a98100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/2570893370 shutdown_connections 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.791+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/2570893370 wait complete. 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 Processor -- start 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 -- start start 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 0x7f4a98198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a98103b80 0x7f4a981990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a98104540 0x7f4a981a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9810e4c0 con 0x7f4a98102990 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9810e340 con 0x7f4a98103b80 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a9e4a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9810e640 con 0x7f4a98104540 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a97fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 0x7f4a98198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a97fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 0x7f4a98198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42820/0 (socket says 172.21.15.1:42820) 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.792+0000 7f4a97fff640 1 -- 172.21.15.1:0/3557766497 learned_addr learned my addr 172.21.15.1:0/3557766497 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a977fe640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a98103b80 0x7f4a981990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a97fff640 1 -- 172.21.15.1:0/3557766497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a98104540 msgr2=0x7f4a981a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a97fff640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a98104540 0x7f4a981a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a97fff640 1 -- 172.21.15.1:0/3557766497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a98103b80 msgr2=0x7f4a981990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a97fff640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a98103b80 0x7f4a981990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a97fff640 1 -- 172.21.15.1:0/3557766497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a981a07c0 con 0x7f4a98102990 2024-04-26T22:49:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a97fff640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 0x7f4a98198ba0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800e9c0 tx=0x7f4a8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8800cd80 con 0x7f4a98102990 2024-04-26T22:49:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a880035f0 con 0x7f4a98102990 2024-04-26T22:49:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a981a0ab0 con 0x7f4a98102990 2024-04-26T22:49:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88010690 con 0x7f4a98102990 2024-04-26T22:49:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.793+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a981a1000 con 0x7f4a98102990 2024-04-26T22:49:33.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.794+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a98109cd0 con 0x7f4a98102990 2024-04-26T22:49:33.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.795+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4a88004000 con 0x7f4a98102990 2024-04-26T22:49:33.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.796+0000 7f4a957fa640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4a800a3cc0 0x7f4a800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:33.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.796+0000 7f4a977fe640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4a800a3cc0 0x7f4a800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:33.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.796+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f4a880d0b10 con 0x7f4a98102990 2024-04-26T22:49:33.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.796+0000 7f4a977fe640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4a800a3cc0 0x7f4a800a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4a9819a0b0 tx=0x7f4a8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:33.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.800+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a8809d3c0 con 0x7f4a98102990 2024-04-26T22:49:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:33 smithi038 ceph-mon[32433]: pgmap v348: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:33 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3578127226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:33 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/4285977365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:33 smithi186 ceph-mon[32774]: pgmap v348: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:33 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3578127226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:33 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/4285977365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:33.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.987+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4a9802dd50 con 0x7f4a98102990 2024-04-26T22:49:33.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.988+0000 7f4a957fa640 1 -- 172.21.15.1:0/3557766497 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f4a8809ccc0 con 0x7f4a98102990 2024-04-26T22:49:33.988 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T22:49:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4a800a3cc0 msgr2=0x7f4a800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4a800a3cc0 0x7f4a800a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4a9819a0b0 tx=0x7f4a8c005f90 comp rx=0 tx=0).stop 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 msgr2=0x7f4a98198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 0x7f4a98198ba0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800e9c0 tx=0x7f4a8800ee80 comp rx=0 tx=0).stop 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 shutdown_connections 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4a800a3cc0 0x7f4a800a6180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a98104540 0x7f4a981a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a98103b80 0x7f4a981990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 --2- 172.21.15.1:0/3557766497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a98102990 0x7f4a98198ba0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 >> 172.21.15.1:0/3557766497 conn(0x7f4a980fdf00 msgr2=0x7f4a98106ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 shutdown_connections 2024-04-26T22:49:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:33.990+0000 7f4a9e4a9640 1 -- 172.21.15.1:0/3557766497 wait complete. 2024-04-26T22:49:34.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:49:34.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.104+0000 7f57ebc93640 1 -- 172.21.15.1:0/849550329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4103470 msgr2=0x7f57e4105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.104+0000 7f57ebc93640 1 --2- 172.21.15.1:0/849550329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4103470 0x7f57e4105860 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009900 tx=0x7f57d802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:34.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- 172.21.15.1:0/849550329 shutdown_connections 2024-04-26T22:49:34.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 --2- 172.21.15.1:0/849550329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4105da0 0x7f57e4108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 --2- 172.21.15.1:0/849550329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57e41086d0 0x7f57e410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 --2- 172.21.15.1:0/849550329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4103470 0x7f57e4105860 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- 172.21.15.1:0/849550329 >> 172.21.15.1:0/849550329 conn(0x7f57e40fce40 msgr2=0x7f57e40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- 172.21.15.1:0/849550329 shutdown_connections 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- 172.21.15.1:0/849550329 wait complete. 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 Processor -- start 2024-04-26T22:49:34.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- start start 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4103470 0x7f57e4102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57e4105da0 0x7f57e4101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 0x7f57e4101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410e530 con 0x7f57e41086d0 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410e3b0 con 0x7f57e4105da0 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ebc93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410e6b0 con 0x7f57e4103470 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.105+0000 7f57ea230640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 0x7f57e4101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 0x7f57e4101960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42840/0 (socket says 172.21.15.1:42840) 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 -- 172.21.15.1:0/2350547144 learned_addr learned my addr 172.21.15.1:0/2350547144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57e922e640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57e4105da0 0x7f57e4101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57e9a2f640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4103470 0x7f57e4102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 -- 172.21.15.1:0/2350547144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4103470 msgr2=0x7f57e4102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4103470 0x7f57e4102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 -- 172.21.15.1:0/2350547144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57e4105da0 msgr2=0x7f57e4101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57e4105da0 0x7f57e4101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 -- 172.21.15.1:0/2350547144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e41021f0 con 0x7f57e41086d0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57e9a2f640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4103470 0x7f57e4102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ea230640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 0x7f57e4101960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00eb10 tx=0x7f57dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004b20 con 0x7f57e41086d0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57dc004cc0 con 0x7f57e41086d0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e41a2d20 con 0x7f57e41086d0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc010670 con 0x7f57e41086d0 2024-04-26T22:49:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.106+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e41a31c0 con 0x7f57e41086d0 2024-04-26T22:49:34.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.107+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e40fe930 con 0x7f57e41086d0 2024-04-26T22:49:34.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.108+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f57dc005020 con 0x7f57e41086d0 2024-04-26T22:49:34.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.109+0000 7f57d6ffd640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57d00a3c40 0x7f57d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.109+0000 7f57e9a2f640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57d00a3c40 0x7f57d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.109+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f57dc0d0bf0 con 0x7f57e41086d0 2024-04-26T22:49:34.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.110+0000 7f57e9a2f640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57d00a3c40 0x7f57d00a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f57d802a930 tx=0x7f57d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:34.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.114+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f57dc09d4a0 con 0x7f57e41086d0 2024-04-26T22:49:34.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.340+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57e402dd20 con 0x7f57e41086d0 2024-04-26T22:49:34.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.340+0000 7f57d6ffd640 1 -- 172.21.15.1:0/2350547144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11917 (secure 0 0 0) 0x7f57dc09cda0 con 0x7f57e41086d0 2024-04-26T22:49:34.341 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:47:47.801060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[24475,24514,24487,24473,14631]},"id":1}]} 2024-04-26T22:49:34.341 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.342+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57d00a3c40 msgr2=0x7f57d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.342+0000 7f57ebc93640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57d00a3c40 0x7f57d00a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f57d802a930 tx=0x7f57d8005fd0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.342+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 msgr2=0x7f57e4101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.342+0000 7f57ebc93640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 0x7f57e4101960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00eb10 tx=0x7f57dc00c2d0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 shutdown_connections 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57d00a3c40 0x7f57d00a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e41086d0 0x7f57e4101960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57e4105da0 0x7f57e4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 --2- 172.21.15.1:0/2350547144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57e4103470 0x7f57e4102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 >> 172.21.15.1:0/2350547144 conn(0x7f57e40fce40 msgr2=0x7f57e4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 shutdown_connections 2024-04-26T22:49:34.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.343+0000 7f57ebc93640 1 -- 172.21.15.1:0/2350547144 wait complete. 2024-04-26T22:49:34.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.456+0000 7f73c7637640 1 -- 172.21.15.1:0/2743638756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0105d40 msgr2=0x7f73c0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.456+0000 7f73c7637640 1 --2- 172.21.15.1:0/2743638756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0105d40 0x7f73c0108130 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009900 tx=0x7f73b002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.456+0000 7f73c7637640 1 -- 172.21.15.1:0/2743638756 shutdown_connections 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.456+0000 7f73c7637640 1 --2- 172.21.15.1:0/2743638756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73c0108670 0x7f73c010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.456+0000 7f73c7637640 1 --2- 172.21.15.1:0/2743638756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0105d40 0x7f73c0108130 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.456+0000 7f73c7637640 1 --2- 172.21.15.1:0/2743638756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73c0103410 0x7f73c0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- 172.21.15.1:0/2743638756 >> 172.21.15.1:0/2743638756 conn(0x7f73c00fce00 msgr2=0x7f73c00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- 172.21.15.1:0/2743638756 shutdown_connections 2024-04-26T22:49:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- 172.21.15.1:0/2743638756 wait complete. 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 Processor -- start 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- start start 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73c0103410 0x7f73c0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73c0105d40 0x7f73c0199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 0x7f73c01a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f73c010e4d0 con 0x7f73c0108670 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f73c010e350 con 0x7f73c0105d40 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c7637640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73c010e650 con 0x7f73c0103410 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c5bd4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 0x7f73c01a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c5bd4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 0x7f73c01a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42852/0 (socket says 172.21.15.1:42852) 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.457+0000 7f73c5bd4640 1 -- 172.21.15.1:0/1413273878 learned_addr learned my addr 172.21.15.1:0/1413273878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c53d3640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73c0103410 0x7f73c0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c5bd4640 1 -- 172.21.15.1:0/1413273878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73c0103410 msgr2=0x7f73c0198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c5bd4640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73c0103410 0x7f73c0198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c5bd4640 1 -- 172.21.15.1:0/1413273878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73c0105d40 msgr2=0x7f73c0199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c4bd2640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73c0105d40 0x7f73c0199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c5bd4640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73c0105d40 0x7f73c0199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c5bd4640 1 -- 172.21.15.1:0/1413273878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c01a0800 con 0x7f73c0108670 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c5bd4640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 0x7f73c01a0190 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00d750 tx=0x7f73bc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc017070 con 0x7f73c0108670 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc0054d0 con 0x7f73c0108670 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c01a0af0 con 0x7f73c0108670 2024-04-26T22:49:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc004c60 con 0x7f73c0108670 2024-04-26T22:49:34.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.458+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c01a0ff0 con 0x7f73c0108670 2024-04-26T22:49:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.459+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c00fe8f0 con 0x7f73c0108670 2024-04-26T22:49:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.461+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f73bc009bd0 con 0x7f73c0108670 2024-04-26T22:49:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.461+0000 7f73b67fc640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f73ac0a3c40 0x7f73ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.461+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f73bc0d0420 con 0x7f73c0108670 2024-04-26T22:49:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.461+0000 7f73c53d3640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f73ac0a3c40 0x7f73ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.462+0000 7f73c53d3640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f73ac0a3c40 0x7f73ac0a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f73b8005fd0 tx=0x7f73b8009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.465+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f73bc09cc70 con 0x7f73c0108670 2024-04-26T22:49:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.652+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73c0063530 con 0x7f73c0108670 2024-04-26T22:49:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.654+0000 7f73b67fc640 1 -- 172.21.15.1:0/1413273878 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f73bc09c570 con 0x7f73c0108670 2024-04-26T22:49:34.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:34.654 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f73ac0a3c40 msgr2=0x7f73ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f73ac0a3c40 0x7f73ac0a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f73b8005fd0 tx=0x7f73b8009870 comp rx=0 tx=0).stop 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 msgr2=0x7f73c01a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 0x7f73c01a0190 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00d750 tx=0x7f73bc00dc10 comp rx=0 tx=0).stop 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 shutdown_connections 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f73ac0a3c40 0x7f73ac0a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73c0108670 0x7f73c01a0190 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73c0105d40 0x7f73c0199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 --2- 172.21.15.1:0/1413273878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73c0103410 0x7f73c0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 >> 172.21.15.1:0/1413273878 conn(0x7f73c00fce00 msgr2=0x7f73c0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 shutdown_connections 2024-04-26T22:49:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.656+0000 7f73c7637640 1 -- 172.21.15.1:0/1413273878 wait complete. 2024-04-26T22:49:34.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2796062723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:34.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3557766497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:34.666 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:34 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2350547144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:34.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T22:47:39.631594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T22:47:40.081969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T22:49:34.668 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T22:49:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 -- 172.21.15.1:0/1858974209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e201045c0 msgr2=0x7f1e2010ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 --2- 172.21.15.1:0/1858974209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e201045c0 0x7f1e2010ae50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009930 tx=0x7f1e1002a420 comp rx=0 tx=0).stop 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 -- 172.21.15.1:0/1858974209 shutdown_connections 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 --2- 172.21.15.1:0/1858974209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e201045c0 0x7f1e2010ae50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 --2- 172.21.15.1:0/1858974209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e20103c00 0x7f1e20104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 --2- 172.21.15.1:0/1858974209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20100c30 0x7f1e20101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 -- 172.21.15.1:0/1858974209 >> 172.21.15.1:0/1858974209 conn(0x7f1e200fc580 msgr2=0x7f1e200fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 -- 172.21.15.1:0/1858974209 shutdown_connections 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 -- 172.21.15.1:0/1858974209 wait complete. 2024-04-26T22:49:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.773+0000 7f1e251f0640 1 Processor -- start 2024-04-26T22:49:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 -- start start 2024-04-26T22:49:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 0x7f1e20198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20103c00 0x7f1e20199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e201045c0 0x7f1e201a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2010e4b0 con 0x7f1e20100c30 2024-04-26T22:49:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2010e330 con 0x7f1e201045c0 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2010e630 con 0x7f1e20103c00 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 0x7f1e20198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 0x7f1e20198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42874/0 (socket says 172.21.15.1:42874) 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 -- 172.21.15.1:0/3645443992 learned_addr learned my addr 172.21.15.1:0/3645443992 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1e59c640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20103c00 0x7f1e20199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1f59e640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e201045c0 0x7f1e201a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 -- 172.21.15.1:0/3645443992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20103c00 msgr2=0x7f1e20199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20103c00 0x7f1e20199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 -- 172.21.15.1:0/3645443992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e201045c0 msgr2=0x7f1e201a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e201045c0 0x7f1e201a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 -- 172.21.15.1:0/3645443992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e201a07e0 con 0x7f1e20100c30 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1f59e640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e201045c0 0x7f1e201a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1e59c640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20103c00 0x7f1e20199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e1ed9d640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 0x7f1e20198c10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800eaa0 tx=0x7f1e1800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e18016070 con 0x7f1e20100c30 2024-04-26T22:49:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e18003e20 con 0x7f1e20100c30 2024-04-26T22:49:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e201a0ad0 con 0x7f1e20100c30 2024-04-26T22:49:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.774+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e180116d0 con 0x7f1e20100c30 2024-04-26T22:49:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.775+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e201a0fd0 con 0x7f1e20100c30 2024-04-26T22:49:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.776+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e20109e50 con 0x7f1e20100c30 2024-04-26T22:49:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.777+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1e18019070 con 0x7f1e20100c30 2024-04-26T22:49:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.777+0000 7f1e17fff640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1e0c0a3c40 0x7f1e0c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.777+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f1e180d06e0 con 0x7f1e20100c30 2024-04-26T22:49:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.777+0000 7f1e1e59c640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1e0c0a3c40 0x7f1e0c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.778+0000 7f1e1e59c640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1e0c0a3c40 0x7f1e0c0a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1e2019a0d0 tx=0x7f1e08005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:34.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.782+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1e1809cf90 con 0x7f1e20100c30 2024-04-26T22:49:34.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:34 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2796062723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:34.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:34 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3557766497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:34.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:34 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2350547144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:34.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.967+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1e20063530 con 0x7f1e20100c30 2024-04-26T22:49:34.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.968+0000 7f1e17fff640 1 -- 172.21.15.1:0/3645443992 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f1e18085e20 con 0x7f1e20100c30 2024-04-26T22:49:34.968 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T22:49:34.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1e0c0a3c40 msgr2=0x7f1e0c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1e0c0a3c40 0x7f1e0c0a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1e2019a0d0 tx=0x7f1e08005ec0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 msgr2=0x7f1e20198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 0x7f1e20198c10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800eaa0 tx=0x7f1e1800ef60 comp rx=0 tx=0).stop 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 shutdown_connections 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f1e0c0a3c40 0x7f1e0c0a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e20103c00 0x7f1e20199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e201045c0 0x7f1e201a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 --2- 172.21.15.1:0/3645443992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20100c30 0x7f1e20198c10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 >> 172.21.15.1:0/3645443992 conn(0x7f1e200fc580 msgr2=0x7f1e20107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 shutdown_connections 2024-04-26T22:49:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:34.970+0000 7f1e251f0640 1 -- 172.21.15.1:0/3645443992 wait complete. 2024-04-26T22:49:34.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T22:49:34.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:34 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2796062723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:34.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:34 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3557766497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:34.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:34 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2350547144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 -- 172.21.15.1:0/4223099662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e8108650 msgr2=0x7f51e810aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/4223099662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e8108650 0x7f51e810aaa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009ca0 tx=0x7f51e002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 -- 172.21.15.1:0/4223099662 shutdown_connections 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/4223099662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e8105d20 0x7f51e8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/4223099662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e8108650 0x7f51e810aaa0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/4223099662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e81033f0 0x7f51e81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 -- 172.21.15.1:0/4223099662 >> 172.21.15.1:0/4223099662 conn(0x7f51e80fce00 msgr2=0x7f51e80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 -- 172.21.15.1:0/4223099662 shutdown_connections 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.087+0000 7f51ed6f1640 1 -- 172.21.15.1:0/4223099662 wait complete. 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 Processor -- start 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 -- start start 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 0x7f51e8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e8105d20 0x7f51e8199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e8108650 0x7f51e81a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810e4a0 con 0x7f51e81033f0 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810e320 con 0x7f51e8105d20 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51ed6f1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810e620 con 0x7f51e8108650 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 0x7f51e8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 0x7f51e8198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42892/0 (socket says 172.21.15.1:42892) 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51e6ffd640 1 -- 172.21.15.1:0/1455050677 learned_addr learned my addr 172.21.15.1:0/1455050677 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.088+0000 7f51e77fe640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e8108650 0x7f51e81a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e67fc640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e8105d20 0x7f51e8199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e6ffd640 1 -- 172.21.15.1:0/1455050677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e8108650 msgr2=0x7f51e81a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e6ffd640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e8108650 0x7f51e81a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e6ffd640 1 -- 172.21.15.1:0/1455050677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e8105d20 msgr2=0x7f51e8199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e6ffd640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e8105d20 0x7f51e8199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e6ffd640 1 -- 172.21.15.1:0/1455050677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e81a0720 con 0x7f51e81033f0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e67fc640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e8105d20 0x7f51e8199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51e6ffd640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 0x7f51e8198b00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f51d800c920 tx=0x7f51d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8007c70 con 0x7f51e81033f0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e81a0a10 con 0x7f51e81033f0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d8007e10 con 0x7f51e81033f0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8018820 con 0x7f51e81033f0 2024-04-26T22:49:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.089+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e81a0fb0 con 0x7f51e81033f0 2024-04-26T22:49:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.090+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e80fe8f0 con 0x7f51e81033f0 2024-04-26T22:49:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.091+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f51d800f450 con 0x7f51e81033f0 2024-04-26T22:49:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.092+0000 7f51cffff640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f51d00a3c40 0x7f51d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.092+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f51d80d0670 con 0x7f51e81033f0 2024-04-26T22:49:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.092+0000 7f51e67fc640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f51d00a3c40 0x7f51d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.092+0000 7f51e67fc640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f51d00a3c40 0x7f51d00a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0058c0 tx=0x7f51dc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.096+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f51d8014030 con 0x7f51e81033f0 2024-04-26T22:49:35.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.280+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f51e802dad0 con 0x7f51e81033f0 2024-04-26T22:49:35.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.281+0000 7f51cffff640 1 -- 172.21.15.1:0/1455050677 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f51d809cf20 con 0x7f51e81033f0 2024-04-26T22:49:35.281 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T22:49:35.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.282+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f51d00a3c40 msgr2=0x7f51d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.282+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f51d00a3c40 0x7f51d00a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0058c0 tx=0x7f51dc005850 comp rx=0 tx=0).stop 2024-04-26T22:49:35.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.282+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 msgr2=0x7f51e8198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.282+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 0x7f51e8198b00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f51d800c920 tx=0x7f51d800cde0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 shutdown_connections 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f51d00a3c40 0x7f51d00a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51e8108650 0x7f51e81a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51e8105d20 0x7f51e8199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 --2- 172.21.15.1:0/1455050677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e81033f0 0x7f51e8198b00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 >> 172.21.15.1:0/1455050677 conn(0x7f51e80fce00 msgr2=0x7f51e8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 shutdown_connections 2024-04-26T22:49:35.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.283+0000 7f51ed6f1640 1 -- 172.21.15.1:0/1455050677 wait complete. 2024-04-26T22:49:35.294 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 -- 172.21.15.1:0/1598817215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c105d20 msgr2=0x7f572c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 --2- 172.21.15.1:0/1598817215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c105d20 0x7f572c108110 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5720009900 tx=0x7f572002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 -- 172.21.15.1:0/1598817215 shutdown_connections 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 --2- 172.21.15.1:0/1598817215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f572c108650 0x7f572c10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 --2- 172.21.15.1:0/1598817215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c105d20 0x7f572c108110 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 --2- 172.21.15.1:0/1598817215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f572c1033f0 0x7f572c1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 -- 172.21.15.1:0/1598817215 >> 172.21.15.1:0/1598817215 conn(0x7f572c0fce00 msgr2=0x7f572c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 -- 172.21.15.1:0/1598817215 shutdown_connections 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.398+0000 7f57319f0640 1 -- 172.21.15.1:0/1598817215 wait complete. 2024-04-26T22:49:35.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 Processor -- start 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 -- start start 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f572c1033f0 0x7f572c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f572c105d20 0x7f572c1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 0x7f572c1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f572c10e490 con 0x7f572c108650 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f572c10e310 con 0x7f572c105d20 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f57319f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f572c10e610 con 0x7f572c1033f0 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f572b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 0x7f572c1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.399+0000 7f572b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 0x7f572c1a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42924/0 (socket says 172.21.15.1:42924) 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 -- 172.21.15.1:0/3803321625 learned_addr learned my addr 172.21.15.1:0/3803321625 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 -- 172.21.15.1:0/3803321625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f572c1033f0 msgr2=0x7f572c198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572a7fc640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f572c105d20 0x7f572c1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572affd640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f572c1033f0 0x7f572c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f572c1033f0 0x7f572c198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 -- 172.21.15.1:0/3803321625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f572c105d20 msgr2=0x7f572c1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f572c105d20 0x7f572c1990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 -- 172.21.15.1:0/3803321625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c1a0790 con 0x7f572c108650 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f572b7fe640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 0x7f572c1a0120 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f572400d950 tx=0x7f572400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724017030 con 0x7f572c108650 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5724004020 con 0x7f572c108650 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c1a0a80 con 0x7f572c108650 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724003b50 con 0x7f572c108650 2024-04-26T22:49:35.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.400+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c1a0f80 con 0x7f572c108650 2024-04-26T22:49:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.401+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572c0fe8f0 con 0x7f572c108650 2024-04-26T22:49:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.402+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f572401a070 con 0x7f572c108650 2024-04-26T22:49:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.403+0000 7f57309ee640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57140a3c40 0x7f57140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.403+0000 7f572affd640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57140a3c40 0x7f57140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.403+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f57240cfe20 con 0x7f572c108650 2024-04-26T22:49:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.403+0000 7f572affd640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57140a3c40 0x7f57140a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f571c008d20 tx=0x7f571c0082b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:35.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.407+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5724021070 con 0x7f572c108650 2024-04-26T22:49:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.593+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f572c063530 con 0x7f572c108650 2024-04-26T22:49:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.594+0000 7f57309ee640 1 -- 172.21.15.1:0/3803321625 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f572409c670 con 0x7f572c108650 2024-04-26T22:49:35.594 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T22:49:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:35 smithi001 ceph-mon[29435]: pgmap v349: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:35 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1413273878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:35 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3645443992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:35 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1455050677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57140a3c40 msgr2=0x7f57140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57140a3c40 0x7f57140a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f571c008d20 tx=0x7f571c0082b0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 msgr2=0x7f572c1a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 0x7f572c1a0120 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f572400d950 tx=0x7f572400de10 comp rx=0 tx=0).stop 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 shutdown_connections 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f57140a3c40 0x7f57140a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f572c108650 0x7f572c1a0120 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f572c105d20 0x7f572c1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 --2- 172.21.15.1:0/3803321625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f572c1033f0 0x7f572c198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 >> 172.21.15.1:0/3803321625 conn(0x7f572c0fce00 msgr2=0x7f572c103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 shutdown_connections 2024-04-26T22:49:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.596+0000 7f57319f0640 1 -- 172.21.15.1:0/3803321625 wait complete. 2024-04-26T22:49:35.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 -- 172.21.15.1:0/3598670321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8103440 msgr2=0x7f40a8109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 --2- 172.21.15.1:0/3598670321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8103440 0x7f40a8109cd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f40a0009ca0 tx=0x7f40a002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 -- 172.21.15.1:0/3598670321 shutdown_connections 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 --2- 172.21.15.1:0/3598670321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40a8102a80 0x7f40a8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 --2- 172.21.15.1:0/3598670321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8103440 0x7f40a8109cd0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 --2- 172.21.15.1:0/3598670321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8101890 0x7f40a8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 -- 172.21.15.1:0/3598670321 >> 172.21.15.1:0/3598670321 conn(0x7f40a80fce40 msgr2=0x7f40a80ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 -- 172.21.15.1:0/3598670321 shutdown_connections 2024-04-26T22:49:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.708+0000 7f40afb66640 1 -- 172.21.15.1:0/3598670321 wait complete. 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 Processor -- start 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 -- start start 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 0x7f40a8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8102a80 0x7f40a8199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40a8103440 0x7f40a81a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f40a810d3c0 con 0x7f40a8101890 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f40a810d240 con 0x7f40a8102a80 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40afb66640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40a810d540 con 0x7f40a8103440 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40ad902640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 0x7f40a8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40ad902640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 0x7f40a8198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42944/0 (socket says 172.21.15.1:42944) 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40ad902640 1 -- 172.21.15.1:0/2759142092 learned_addr learned my addr 172.21.15.1:0/2759142092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.709+0000 7f40ad101640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8102a80 0x7f40a8199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad902640 1 -- 172.21.15.1:0/2759142092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40a8103440 msgr2=0x7f40a81a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad902640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40a8103440 0x7f40a81a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad902640 1 -- 172.21.15.1:0/2759142092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8102a80 msgr2=0x7f40a8199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad902640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8102a80 0x7f40a8199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad902640 1 -- 172.21.15.1:0/2759142092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a81a06e0 con 0x7f40a8101890 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad101640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8102a80 0x7f40a8199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40ad902640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 0x7f40a8198ac0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f40a400c970 tx=0x7f40a400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4007c10 con 0x7f40a8101890 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a81a09d0 con 0x7f40a8101890 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a4007db0 con 0x7f40a8101890 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a400f7d0 con 0x7f40a8101890 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.710+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a81a0f20 con 0x7f40a8101890 2024-04-26T22:49:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.711+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a81a11b0 con 0x7f40a8101890 2024-04-26T22:49:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.712+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f40a4018480 con 0x7f40a8101890 2024-04-26T22:49:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.713+0000 7f409effd640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40940a3c40 0x7f40940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.713+0000 7f40ad101640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40940a3c40 0x7f40940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.713+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f40a40d0bd0 con 0x7f40a8101890 2024-04-26T22:49:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.713+0000 7f40ad101640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40940a3c40 0x7f40940a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f40a8199fd0 tx=0x7f4098005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:35.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.716+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f40a409d540 con 0x7f40a8101890 2024-04-26T22:49:35.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:35 smithi038 ceph-mon[32433]: pgmap v349: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:35.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:35 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1413273878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:35.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:35 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3645443992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:35.853 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:35 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1455050677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:35.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.942+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40a81a1380 con 0x7f40a8101890 2024-04-26T22:49:35.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.943+0000 7f409effd640 1 -- 172.21.15.1:0/2759142092 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11917 (secure 0 0 0) 0x7f40a409ce40 con 0x7f40a8101890 2024-04-26T22:49:35.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:35.944 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:47:47.801060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[24475,24514,24487,24473,14631]},"id":1}]} 2024-04-26T22:49:35.944 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-04-26T22:49:35.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40940a3c40 msgr2=0x7f40940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40940a3c40 0x7f40940a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f40a8199fd0 tx=0x7f4098005d30 comp rx=0 tx=0).stop 2024-04-26T22:49:35.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 msgr2=0x7f40a8198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:35.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 0x7f40a8198ac0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f40a400c970 tx=0x7f40a400ce30 comp rx=0 tx=0).stop 2024-04-26T22:49:35.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 shutdown_connections 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f40940a3c40 0x7f40940a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40a8103440 0x7f40a81a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a8102a80 0x7f40a8199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 --2- 172.21.15.1:0/2759142092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a8101890 0x7f40a8198ac0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 >> 172.21.15.1:0/2759142092 conn(0x7f40a80fce40 msgr2=0x7f40a80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 shutdown_connections 2024-04-26T22:49:35.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:35.946+0000 7f40afb66640 1 -- 172.21.15.1:0/2759142092 wait complete. 2024-04-26T22:49:35.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T22:49:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:35 smithi186 ceph-mon[32774]: pgmap v349: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:35 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1413273878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:35 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3645443992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T22:49:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:35 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1455050677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T22:49:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 -- 172.21.15.1:0/948460035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38104540 msgr2=0x7f4f3810add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 --2- 172.21.15.1:0/948460035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38104540 0x7f4f3810add0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009ca0 tx=0x7f4f3002a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 -- 172.21.15.1:0/948460035 shutdown_connections 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 --2- 172.21.15.1:0/948460035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f38103b80 0x7f4f38104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 --2- 172.21.15.1:0/948460035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38104540 0x7f4f3810add0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 --2- 172.21.15.1:0/948460035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f38102990 0x7f4f38102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 -- 172.21.15.1:0/948460035 >> 172.21.15.1:0/948460035 conn(0x7f4f380fdf20 msgr2=0x7f4f38100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 -- 172.21.15.1:0/948460035 shutdown_connections 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 -- 172.21.15.1:0/948460035 wait complete. 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 Processor -- start 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.059+0000 7f4f3e623640 1 -- start start 2024-04-26T22:49:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3e623640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f38102990 0x7f4f38198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3e623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 0x7f4f38199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3e623640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f38104540 0x7f4f381a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3e623640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3810e520 con 0x7f4f38103b80 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3e623640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3810e3a0 con 0x7f4f38104540 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3e623640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3810e6a0 con 0x7f4f38102990 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 0x7f4f38199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 0x7f4f38199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42962/0 (socket says 172.21.15.1:42962) 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 -- 172.21.15.1:0/3820876505 learned_addr learned my addr 172.21.15.1:0/3820876505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f37fff640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f38102990 0x7f4f38198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f3cbc0640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f38104540 0x7f4f381a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 -- 172.21.15.1:0/3820876505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f38102990 msgr2=0x7f4f38198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f38102990 0x7f4f38198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 -- 172.21.15.1:0/3820876505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f38104540 msgr2=0x7f4f381a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f38104540 0x7f4f381a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 -- 172.21.15.1:0/3820876505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f381a07e0 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f377fe640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 0x7f4f38199100 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00b4a0 tx=0x7f4f2c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.060+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c014070 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.061+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c010ac0 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.061+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c017c30 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.061+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f381a0ad0 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.061+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f381a1020 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.062+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f38109cb0 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.063+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4f2c017070 con 0x7f4f38103b80 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.064+0000 7f4f357fa640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f200a3cc0 0x7f4f200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.064+0000 7f4f37fff640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f200a3cc0 0x7f4f200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.064+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f4f2c0cf890 con 0x7f4f38103b80 2024-04-26T22:49:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.064+0000 7f4f37fff640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f200a3cc0 0x7f4f200a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f28005fd0 tx=0x7f4f28005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.068+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4f2c09c1f0 con 0x7f4f38103b80 2024-04-26T22:49:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.257+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f3802dcf0 con 0x7f4f38103b80 2024-04-26T22:49:36.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.258+0000 7f4f357fa640 1 -- 172.21.15.1:0/3820876505 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f4f2c09baf0 con 0x7f4f38103b80 2024-04-26T22:49:36.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T22:49:36.259 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f200a3cc0 msgr2=0x7f4f200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f200a3cc0 0x7f4f200a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f28005fd0 tx=0x7f4f28005850 comp rx=0 tx=0).stop 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 msgr2=0x7f4f38199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 0x7f4f38199100 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00b4a0 tx=0x7f4f2c00b960 comp rx=0 tx=0).stop 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 shutdown_connections 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f4f200a3cc0 0x7f4f200a6180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f38104540 0x7f4f381a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f38103b80 0x7f4f38199100 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 --2- 172.21.15.1:0/3820876505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f38102990 0x7f4f38198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 >> 172.21.15.1:0/3820876505 conn(0x7f4f380fdf20 msgr2=0x7f4f38106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 shutdown_connections 2024-04-26T22:49:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.260+0000 7f4f3e623640 1 -- 172.21.15.1:0/3820876505 wait complete. 2024-04-26T22:49:36.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T22:47:41.130042+0000', 'modified': '2024-04-26T22:47:47.801060+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24514, 'mds_1': 24487, 'mds_2': 24475, 'mds_3': 24473, 'mds_4': 14631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.1:6839/1512535471', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6838', 'nonce': 1512535471}, {'type': 'v1', 'addr': '172.21.15.1:6839', 'nonce': 1512535471}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 4, 'incarnation': 23, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6841/1622627716', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6840', 'nonce': 1622627716}, {'type': 'v1', 'addr': '172.21.15.1:6841', 'nonce': 1622627716}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14643': {'gid': 14643, 'name': 'b', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6839/2490292790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 2490292790}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 2490292790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6833/3943944853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6832', 'nonce': 3943944853}, {'type': 'v1', 'addr': '172.21.15.38:6833', 'nonce': 3943944853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6833/4165972801', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6832', 'nonce': 4165972801}, {'type': 'v1', 'addr': '172.21.15.186:6833', 'nonce': 4165972801}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6835/1362256204', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6834', 'nonce': 1362256204}, {'type': 'v1', 'addr': '172.21.15.38:6835', 'nonce': 1362256204}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 3, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.38:6837/1812912662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6836', 'nonce': 1812912662}, {'type': 'v1', 'addr': '172.21.15.38:6837', 'nonce': 1812912662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6835/1379199020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 1379199020}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 1379199020}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.186:6837/786403012', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 786403012}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 786403012}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.186:6839/3709434074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 3709434074}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 3709434074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24514, 'qdb_cluster': [24475, 24514, 24487, 24473, 14631]} 2024-04-26T22:49:36.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-26T22:49:36.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 -- 172.21.15.1:0/3359396224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c102a60 msgr2=0x7fed9c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 --2- 172.21.15.1:0/3359396224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c102a60 0x7fed9c102ee0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009900 tx=0x7fed8c02a410 comp rx=0 tx=0).stop 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 -- 172.21.15.1:0/3359396224 shutdown_connections 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 --2- 172.21.15.1:0/3359396224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed9c103420 0x7fed9c109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 --2- 172.21.15.1:0/3359396224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c102a60 0x7fed9c102ee0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 --2- 172.21.15.1:0/3359396224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed9c101870 0x7fed9c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 -- 172.21.15.1:0/3359396224 >> 172.21.15.1:0/3359396224 conn(0x7fed9c0fce00 msgr2=0x7fed9c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 -- 172.21.15.1:0/3359396224 shutdown_connections 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.377+0000 7feda38a0640 1 -- 172.21.15.1:0/3359396224 wait complete. 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 Processor -- start 2024-04-26T22:49:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 -- start start 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed9c101870 0x7fed9c198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed9c102a60 0x7fed9c199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 0x7fed9c1a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c10d3a0 con 0x7fed9c103420 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c10d220 con 0x7fed9c101870 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda38a0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c10d520 con 0x7fed9c102a60 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda1e3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 0x7fed9c1a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda1e3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 0x7fed9c1a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42974/0 (socket says 172.21.15.1:42974) 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda1e3d640 1 -- 172.21.15.1:0/3807302112 learned_addr learned my addr 172.21.15.1:0/3807302112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda163c640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed9c101870 0x7fed9c198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.378+0000 7feda0e3b640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed9c102a60 0x7fed9c199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda1e3d640 1 -- 172.21.15.1:0/3807302112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed9c102a60 msgr2=0x7fed9c199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda1e3d640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed9c102a60 0x7fed9c199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda1e3d640 1 -- 172.21.15.1:0/3807302112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed9c101870 msgr2=0x7fed9c198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda1e3d640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed9c101870 0x7fed9c198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda1e3d640 1 -- 172.21.15.1:0/3807302112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9c1a08b0 con 0x7fed9c103420 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda1e3d640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 0x7fed9c1a0210 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fed9800e970 tx=0x7fed9800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7fed927fc640 1 -- 172.21.15.1:0/3807302112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9800ccc0 con 0x7fed9c103420 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9c1a0ba0 con 0x7fed9c103420 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7fed927fc640 1 -- 172.21.15.1:0/3807302112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed9800ce60 con 0x7fed9c103420 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7fed927fc640 1 -- 172.21.15.1:0/3807302112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed98010430 con 0x7fed9c103420 2024-04-26T22:49:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.379+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9c1a10f0 con 0x7fed9c103420 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.381+0000 7fed927fc640 1 -- 172.21.15.1:0/3807302112 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fed98004ff0 con 0x7fed9c103420 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.381+0000 7feda65bf740 1 Processor -- start 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.381+0000 7feda65bf740 1 -- start start 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda65bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c104aa0 0x55d96c104ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda65bf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 0x55d96c109f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda65bf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c10ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda65bf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d96c065740 con 0x55d96c105490 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda65bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d96c0013a0 con 0x55d96c104aa0 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda65bf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d96c061700 con 0x55d96c10a600 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 0x55d96c109f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 0x55d96c109f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42978/0 (socket says 172.21.15.1:42978) 2024-04-26T22:49:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 -- 172.21.15.1:0/1344739487 learned_addr learned my addr 172.21.15.1:0/1344739487 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c10ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7fed927fc640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fed880a3c40 0x7fed880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c104aa0 0x55d96c104ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 -- 172.21.15.1:0/1344739487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 msgr2=0x55d96c10ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c10ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 -- 172.21.15.1:0/1344739487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c104aa0 msgr2=0x55d96c104ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c104aa0 0x55d96c104ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 -- 172.21.15.1:0/1344739487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96c01d750 con 0x55d96c105490 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda1e3d640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fed880a3c40 0x7fed880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c104aa0 0x55d96c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7fed927fc640 1 -- 172.21.15.1:0/3807302112 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fed980d0a00 con 0x7fed9c103420 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7feda163c640 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 0x55d96c109f90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fed9400aed0 tx=0x7fed9400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c776a75adff30cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.382+0000 7fed877fe640 1 -- 172.21.15.1:0/1344739487 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed94004d20 con 0x55d96c105490 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7fed877fe640 1 -- 172.21.15.1:0/1344739487 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed94010af0 con 0x55d96c105490 2024-04-26T22:49:36.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7fed877fe640 1 -- 172.21.15.1:0/1344739487 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed94005790 con 0x55d96c105490 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 -- 172.21.15.1:0/1344739487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 msgr2=0x55d96c109f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda1e3d640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fed880a3c40 0x7fed880a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed98099580 tx=0x7fed9806e640 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 0x55d96c109f90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fed9400aed0 tx=0x7fed9400f790 comp rx=0 tx=0).stop 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 -- 172.21.15.1:0/1344739487 shutdown_connections 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c10ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c105490 0x55d96c109f90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 --2- 172.21.15.1:0/1344739487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c104aa0 0x55d96c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 -- 172.21.15.1:0/1344739487 >> 172.21.15.1:0/1344739487 conn(0x55d96c0fbd00 msgr2=0x55d96c0fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 -- 172.21.15.1:0/1344739487 shutdown_connections 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.383+0000 7feda65bf740 1 -- 172.21.15.1:0/1344739487 wait complete. 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 -- start start 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 Processor -- start 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c10eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c10f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c116440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d96c10d5b0 con 0x55d96c104aa0 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d96bfecfe0 con 0x55d96c105490 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d96c10d730 con 0x55d96c10a600 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c10eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda65bf740 10 client.0 _ll_register_callbacks cb 0x55d96c003150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c10eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42994/0 (socket says 172.21.15.1:42994) 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 -- 172.21.15.1:0/1738144839 learned_addr learned my addr 172.21.15.1:0/1738144839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda163c640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c10f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 -- 172.21.15.1:0/1738144839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 msgr2=0x55d96c116440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c116440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 -- 172.21.15.1:0/1738144839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 msgr2=0x55d96c10f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c10f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.384+0000 7feda0e3b640 1 -- 172.21.15.1:0/1738144839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96bd772d0 con 0x55d96c104aa0 2024-04-26T22:49:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda163c640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c10eec0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fed8c005940 tx=0x7fed8c005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10b9aaf3d859a17 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7fed84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7fed84ff9640 1 -- 172.21.15.1:0/1738144839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c044070 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7fed84ff9640 1 -- 172.21.15.1:0/1738144839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fed8c004450 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed94010af0 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7fed84ff9640 1 -- 172.21.15.1:0/1738144839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fed8c050050 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 10 client.15252 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d96be416b0 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7fed84ff9640 1 -- 172.21.15.1:0/1738144839 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed94010af0 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7fed84ff9640 1 -- 172.21.15.1:0/1738144839 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fed8c03a040 con 0x55d96c104aa0 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 10 client.15252 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 10 client.15252 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 20 client.15252 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c11b8c0 0x55d96c11cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 4 client.15252 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.385+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 --> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d96c073350 con 0x55d96c11b8c0 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.386+0000 7feda163c640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c11b8c0 0x55d96c11cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.386+0000 7feda163c640 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c11b8c0 0x55d96c11cd30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9803c0745c93a6a6 server_cookie=30df0988c25a9208 in_seq=0 out_seq=0 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.386+0000 7fed84ff9640 10 client.15252 ms_handle_connect on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.386+0000 7fed84ff9640 1 -- 172.21.15.1:0/1738144839 <== mds.4 v2:172.21.15.1:6840/1622627716 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d96c073350 con 0x55d96c11b8c0 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.386+0000 7fed84ff9640 10 client.15252 handle_command_reply: tid=0 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.386+0000 7feda65bf740 1 client.15252 shutdown 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 msgr2=0x55d96c10eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c10eec0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fed8c005940 tx=0x7fed8c005e60 comp rx=0 tx=0).stop 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 shutdown_connections 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c11b8c0 0x55d96c11cd30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c10a600 0x55d96c116440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7fed84ff9640 0 client.15252 ms_handle_reset on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 --2- 172.21.15.1:0/1738144839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c10eec0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 >> 172.21.15.1:0/1738144839 conn(0x55d96c0fbd00 msgr2=0x55d96c0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 shutdown_connections 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 1 -- 172.21.15.1:0/1738144839 wait complete. 2024-04-26T22:49:36.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.387+0000 7feda65bf740 20 client.15252 trim_cache size 0 max 16384 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 Processor -- start 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 -- start start 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c104aa0 0x55d96c106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c116400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 0x55d96c106d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d96c11b070 con 0x55d96c10a600 2024-04-26T22:49:36.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d96c11b740 con 0x55d96c105490 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda65bf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d96c11be10 con 0x55d96c104aa0 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 0x55d96c106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 0x55d96c106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43010/0 (socket says 172.21.15.1:43010) 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 -- 172.21.15.1:0/1252022720 learned_addr learned my addr 172.21.15.1:0/1252022720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c104aa0 0x55d96c106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda163c640 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c116400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 -- 172.21.15.1:0/1252022720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c104aa0 msgr2=0x55d96c106850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c104aa0 0x55d96c106850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 -- 172.21.15.1:0/1252022720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 msgr2=0x55d96c116400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c116400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda0e3b640 1 -- 172.21.15.1:0/1252022720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96c109b00 con 0x55d96c10a600 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.405+0000 7feda163c640 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c116400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda0e3b640 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 0x55d96c106d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fed8c031500 tx=0x7fed8c031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4b04f0fb3a8634 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7fed86ffd640 1 -- 172.21.15.1:0/1252022720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c059070 con 0x55d96c10a600 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7fed86ffd640 1 -- 172.21.15.1:0/1252022720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed8c03d910 con 0x55d96c10a600 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7fed86ffd640 1 -- 172.21.15.1:0/1252022720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c044070 con 0x55d96c10a600 2024-04-26T22:49:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 -- 172.21.15.1:0/1252022720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 msgr2=0x55d96c106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 0x55d96c106d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fed8c031500 tx=0x7fed8c031a10 comp rx=0 tx=0).stop 2024-04-26T22:49:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 -- 172.21.15.1:0/1252022720 shutdown_connections 2024-04-26T22:49:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c10a600 0x55d96c106d90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c105490 0x55d96c116400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 --2- 172.21.15.1:0/1252022720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c104aa0 0x55d96c106850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 -- 172.21.15.1:0/1252022720 >> 172.21.15.1:0/1252022720 conn(0x55d96c0fbd00 msgr2=0x55d96c0fc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 -- 172.21.15.1:0/1252022720 shutdown_connections 2024-04-26T22:49:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.406+0000 7feda65bf740 1 -- 172.21.15.1:0/1252022720 wait complete. 2024-04-26T22:49:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 -- start start 2024-04-26T22:49:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 Processor -- start 2024-04-26T22:49:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c112630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c105490 0x55d96c112b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c10a600 0x55d96c115650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d96c11cbb0 con 0x55d96c104aa0 2024-04-26T22:49:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d96c11c4e0 con 0x55d96c10a600 2024-04-26T22:49:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d96c11d280 con 0x55d96c105490 2024-04-26T22:49:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda65bf740 10 client.0 _ll_register_callbacks cb 0x7fed98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c112630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c112630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43014/0 (socket says 172.21.15.1:43014) 2024-04-26T22:49:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 -- 172.21.15.1:0/2706103826 learned_addr learned my addr 172.21.15.1:0/2706103826 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda163c640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c105490 0x55d96c112b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 -- 172.21.15.1:0/2706103826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c105490 msgr2=0x55d96c112b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c105490 0x55d96c112b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 -- 172.21.15.1:0/2706103826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c10a600 msgr2=0x55d96c115650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c10a600 0x55d96c115650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda0e3b640 1 -- 172.21.15.1:0/2706103826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96c115c60 con 0x55d96c104aa0 2024-04-26T22:49:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.407+0000 7feda163c640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c105490 0x55d96c112b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda0e3b640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c112630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fed8c057040 tx=0x7fed8c005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91e7c4cbc6dad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7fed84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7fed84ff9640 1 -- 172.21.15.1:0/2706103826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c059040 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7fed84ff9640 1 -- 172.21.15.1:0/2706103826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fed8c004250 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed8c03d910 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7fed84ff9640 1 -- 172.21.15.1:0/2706103826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fed8c065070 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 10 client.15258 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d96c181db0 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7fed84ff9640 1 -- 172.21.15.1:0/2706103826 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed8c03d910 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7fed84ff9640 1 -- 172.21.15.1:0/2706103826 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fed8c0035f0 con 0x55d96c104aa0 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 10 client.15258 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 10 client.15258 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 20 client.15258 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c186530 0x55d96c185240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 4 client.15258 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 --> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] -- command(tid 0: {"prefix": "status"}) -- 0x55d96c06d1c0 con 0x55d96c186530 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.408+0000 7feda163c640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c186530 0x55d96c185240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7fed84ff9640 10 client.15258 ms_handle_connect on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda163c640 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c186530 0x55d96c185240 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e923b8afabd8f282 server_cookie=fe0b35f82bdf9a0e in_seq=0 out_seq=0 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7fed84ff9640 1 -- 172.21.15.1:0/2706103826 <== mds.4 v2:172.21.15.1:6840/1622627716 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d96c06d1c0 con 0x55d96c186530 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7fed84ff9640 10 client.15258 handle_command_reply: tid=0 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 client.15258 shutdown 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 msgr2=0x55d96c112630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c112630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fed8c057040 tx=0x7fed8c005e60 comp rx=0 tx=0).stop 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 shutdown_connections 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:6840/1622627716,v1:172.21.15.1:6841/1622627716] conn(0x55d96c186530 0x55d96c185240 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d96c105490 0x55d96c112b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7fed84ff9640 0 client.15258 ms_handle_reset on v2:172.21.15.1:6840/1622627716 2024-04-26T22:49:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d96c10a600 0x55d96c115650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 --2- 172.21.15.1:0/2706103826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d96c104aa0 0x55d96c112630 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 >> 172.21.15.1:0/2706103826 conn(0x55d96c0fbd00 msgr2=0x55d96c106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 shutdown_connections 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 1 -- 172.21.15.1:0/2706103826 wait complete. 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.409+0000 7feda65bf740 20 client.15258 trim_cache size 0 max 16384 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 4, 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 14631, 2024-04-26T22:49:36.423 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 109.61282842200001, 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 123.645678605 2024-04-26T22:49:36.424 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fed880a3c40 msgr2=0x7fed880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fed880a3c40 0x7fed880a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed98099580 tx=0x7fed9806e640 comp rx=0 tx=0).stop 2024-04-26T22:49:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 msgr2=0x7fed9c1a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 0x7fed9c1a0210 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fed9800e970 tx=0x7fed9800ed70 comp rx=0 tx=0).stop 2024-04-26T22:49:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 shutdown_connections 2024-04-26T22:49:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fed880a3c40 0x7fed880a6100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed9c102a60 0x7fed9c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed9c103420 0x7fed9c1a0210 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 --2- 172.21.15.1:0/3807302112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed9c101870 0x7fed9c198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 >> 172.21.15.1:0/3807302112 conn(0x7fed9c0fce00 msgr2=0x7fed9c069ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 shutdown_connections 2024-04-26T22:49:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.410+0000 7feda38a0640 1 -- 172.21.15.1:0/3807302112 wait complete. 2024-04-26T22:49:36.428 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.534+0000 7fceeaa93640 1 -- 172.21.15.1:0/3723427603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4101870 msgr2=0x7fcee4101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.534+0000 7fceeaa93640 1 --2- 172.21.15.1:0/3723427603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4101870 0x7fcee4101c90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fced8009900 tx=0x7fced802a410 comp rx=0 tx=0).stop 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.534+0000 7fceeaa93640 1 -- 172.21.15.1:0/3723427603 shutdown_connections 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.534+0000 7fceeaa93640 1 --2- 172.21.15.1:0/3723427603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4102a60 0x7fcee4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.534+0000 7fceeaa93640 1 --2- 172.21.15.1:0/3723427603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcee4103420 0x7fcee4109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.534+0000 7fceeaa93640 1 --2- 172.21.15.1:0/3723427603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4101870 0x7fcee4101c90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- 172.21.15.1:0/3723427603 >> 172.21.15.1:0/3723427603 conn(0x7fcee40fce00 msgr2=0x7fcee40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- 172.21.15.1:0/3723427603 shutdown_connections 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- 172.21.15.1:0/3723427603 wait complete. 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 Processor -- start 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- start start 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcee4101870 0x7fcee4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 0x7fcee4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4103420 0x7fcee419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcee410d370 con 0x7fcee4102a60 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcee410d1f0 con 0x7fcee4101870 2024-04-26T22:49:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fceeaa93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcee410d4f0 con 0x7fcee4103420 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fcee3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 0x7fcee4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fcee3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 0x7fcee4198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43034/0 (socket says 172.21.15.1:43034) 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fcee3fff640 1 -- 172.21.15.1:0/741865441 learned_addr learned my addr 172.21.15.1:0/741865441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.535+0000 7fcee882f640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcee4101870 0x7fcee4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee9030640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4103420 0x7fcee419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee3fff640 1 -- 172.21.15.1:0/741865441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4103420 msgr2=0x7fcee419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee3fff640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4103420 0x7fcee419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee3fff640 1 -- 172.21.15.1:0/741865441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcee4101870 msgr2=0x7fcee4197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee3fff640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcee4101870 0x7fcee4197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee3fff640 1 -- 172.21.15.1:0/741865441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee419f720 con 0x7fcee4102a60 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee9030640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4103420 0x7fcee419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee3fff640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 0x7fcee4198040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fced000e9f0 tx=0x7fced000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee1ffb640 1 -- 172.21.15.1:0/741865441 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced000cd80 con 0x7fcee4102a60 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee419fa10 con 0x7fcee4102a60 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee1ffb640 1 -- 172.21.15.1:0/741865441 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fced00108e0 con 0x7fcee4102a60 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fcee1ffb640 1 -- 172.21.15.1:0/741865441 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0018980 con 0x7fcee4102a60 2024-04-26T22:49:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.536+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee419ffb0 con 0x7fcee4102a60 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.538+0000 7fcee1ffb640 1 -- 172.21.15.1:0/741865441 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fced0010430 con 0x7fcee4102a60 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 Processor -- start 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 -- start start 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80a6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d7c8003740 con 0x55d7c809f080 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7c7f9f3a0 con 0x55d7c80a41f0 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fceed7b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d7c7fff700 con 0x55d7c809e690 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee1ffb640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fced40a3c40 0x7fced40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43050/0 (socket says 172.21.15.1:43050) 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 -- 172.21.15.1:0/3505896218 learned_addr learned my addr 172.21.15.1:0/3505896218 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee3fff640 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee9030640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fced40a3c40 0x7fced40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee9030640 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee1ffb640 1 -- 172.21.15.1:0/741865441 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fced00d08c0 con 0x7fcee4102a60 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 -- 172.21.15.1:0/3505896218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 msgr2=0x55d7c809eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 -- 172.21.15.1:0/3505896218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 msgr2=0x55d7c80a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.539+0000 7fcee882f640 1 -- 172.21.15.1:0/3505896218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7c7fbb750 con 0x55d7c809f080 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fcee9030640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fced40a3c40 0x7fced40a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55d7c809f8e0 tx=0x7fcedc0098e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fcee882f640 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a3b80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fced8009900 tx=0x7fced8005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472fea1198a1ed2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fceceffd640 1 -- 172.21.15.1:0/3505896218 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8035070 con 0x55d7c809f080 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fceceffd640 1 -- 172.21.15.1:0/3505896218 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fced8031c50 con 0x55d7c809f080 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fceceffd640 1 -- 172.21.15.1:0/3505896218 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8003850 con 0x55d7c809f080 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fceed7b3740 1 -- 172.21.15.1:0/3505896218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 msgr2=0x55d7c80a3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.540+0000 7fceed7b3740 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a3b80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fced8009900 tx=0x7fced8005d70 comp rx=0 tx=0).stop 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- 172.21.15.1:0/3505896218 shutdown_connections 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a3b80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 --2- 172.21.15.1:0/3505896218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- 172.21.15.1:0/3505896218 >> 172.21.15.1:0/3505896218 conn(0x55d7c8099d00 msgr2=0x55d7c809a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- 172.21.15.1:0/3505896218 shutdown_connections 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- 172.21.15.1:0/3505896218 wait complete. 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- start start 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 Processor -- start 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c80a8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80a7140 con 0x55d7c809f080 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7c7f8afe0 con 0x55d7c80a41f0 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80a72c0 con 0x55d7c809e690 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fceed7b3740 10 client.0 _ll_register_callbacks cb 0x55d7c7fa1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.541+0000 7fcee3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43060/0 (socket says 172.21.15.1:43060) 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 -- 172.21.15.1:0/107714820 learned_addr learned my addr 172.21.15.1:0/107714820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee882f640 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c80a8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 -- 172.21.15.1:0/107714820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 msgr2=0x55d7c80a8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c80a8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 -- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 msgr2=0x55d7c80b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 -- 172.21.15.1:0/107714820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7c7d152d0 con 0x55d7c809f080 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcee3fff640 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a8fd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fced009c770 tx=0x7fced006e5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff9d5a3968c1674 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcec3fff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcec3fff640 1 -- 172.21.15.1:0/107714820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced009ca70 con 0x55d7c809f080 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcec3fff640 1 -- 172.21.15.1:0/107714820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fced00a5250 con 0x55d7c809f080 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced8031c50 con 0x55d7c809f080 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcec3fff640 1 -- 172.21.15.1:0/107714820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fced00a0e20 con 0x55d7c809f080 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fceed7b3740 10 client.15270 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d7c7ddf6b0 con 0x55d7c809f080 2024-04-26T22:49:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.542+0000 7fcec3fff640 1 -- 172.21.15.1:0/107714820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fced8031c50 con 0x55d7c809f080 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fcec3fff640 1 -- 172.21.15.1:0/107714820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fced009c050 con 0x55d7c809f080 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fceed7b3740 10 client.15270 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fceed7b3740 10 client.15270 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fceed7b3740 20 client.15270 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fceed7b3740 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c80b5520 0x55d7c80b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fceed7b3740 4 client.15270 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 --> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7c8011350 con 0x55d7c80b5520 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fcee882f640 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c80b5520 0x55d7c80b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fcee882f640 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c80b5520 0x55d7c80b6990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=629d1ab5661604b7 server_cookie=ddb64e16e0abcb8f in_seq=0 out_seq=0 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.543+0000 7fcec3fff640 10 client.15270 ms_handle_connect on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:36.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fcec3fff640 1 -- 172.21.15.1:0/107714820 <== mds.3 v2:172.21.15.38:6836/1812912662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7c8011350 con 0x55d7c80b5520 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fcec3fff640 10 client.15270 handle_command_reply: tid=0 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 client.15270 shutdown 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 msgr2=0x55d7c80a8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a8fd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fced009c770 tx=0x7fced006e5c0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 shutdown_connections 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c80b5520 0x55d7c80b6990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c80a41f0 0x55d7c80b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809f080 0x55d7c80a8fd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fcec3fff640 0 client.15270 ms_handle_reset on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 --2- 172.21.15.1:0/107714820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c80a8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 >> 172.21.15.1:0/107714820 conn(0x55d7c8099d00 msgr2=0x55d7c8098f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 shutdown_connections 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 1 -- 172.21.15.1:0/107714820 wait complete. 2024-04-26T22:49:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.544+0000 7fceed7b3740 20 client.15270 trim_cache size 0 max 16384 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 Processor -- start 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 -- start start 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c80a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 0x55d7c80a82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80b4cd0 con 0x55d7c80a41f0 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80b53a0 con 0x55d7c809f080 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fceed7b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80b5a70 con 0x55d7c809e690 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 0x55d7c80a82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 0x55d7c80a82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43062/0 (socket says 172.21.15.1:43062) 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 -- 172.21.15.1:0/4208255054 learned_addr learned my addr 172.21.15.1:0/4208255054 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 -- 172.21.15.1:0/4208255054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 msgr2=0x55d7c809a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 -- 172.21.15.1:0/4208255054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 msgr2=0x55d7c80a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c80a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 -- 172.21.15.1:0/4208255054 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7c809f910 con 0x55d7c80a41f0 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcee9030640 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 0x55d7c80a82c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0109f0 tx=0x7fcedc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20fcce7ee6f6b6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.563+0000 7fcece7fc640 1 -- 172.21.15.1:0/4208255054 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc02d070 con 0x55d7c80a41f0 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fcece7fc640 1 -- 172.21.15.1:0/4208255054 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcedc010df0 con 0x55d7c80a41f0 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fcece7fc640 1 -- 172.21.15.1:0/4208255054 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0289d0 con 0x55d7c80a41f0 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 -- 172.21.15.1:0/4208255054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 msgr2=0x55d7c80a82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 0x55d7c80a82c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0109f0 tx=0x7fcedc004ab0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 -- 172.21.15.1:0/4208255054 shutdown_connections 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c80a41f0 0x55d7c80a82c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c80a7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4208255054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c809e690 0x55d7c809a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 -- 172.21.15.1:0/4208255054 >> 172.21.15.1:0/4208255054 conn(0x55d7c8099d00 msgr2=0x55d7c80a04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 -- 172.21.15.1:0/4208255054 shutdown_connections 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 -- 172.21.15.1:0/4208255054 wait complete. 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 -- start start 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.564+0000 7fceed7b3740 1 Processor -- start 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 0x55d7c8117ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c8117ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c80a41f0 0x55d7c811f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80b6810 con 0x55d7c809e690 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80b6140 con 0x55d7c809f080 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d7c80b6ee0 con 0x55d7c80a41f0 2024-04-26T22:49:36.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fceed7b3740 10 client.0 _ll_register_callbacks cb 0x7fced000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 0x55d7c8117ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 0x55d7c8117ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43068/0 (socket says 172.21.15.1:43068) 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 -- 172.21.15.1:0/4153503249 learned_addr learned my addr 172.21.15.1:0/4153503249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee3fff640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c8117ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 -- 172.21.15.1:0/4153503249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c80a41f0 msgr2=0x55d7c811f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c80a41f0 0x55d7c811f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 -- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 msgr2=0x55d7c8117ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c8117ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 -- 172.21.15.1:0/4153503249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7c811f620 con 0x55d7c809e690 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee3fff640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c8117ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcee882f640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 0x55d7c8117ab0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fced8005c40 tx=0x7fced80314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db61c81873abd21a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.565+0000 7fcec3fff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fcec3fff640 1 -- 172.21.15.1:0/4153503249 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8035040 con 0x55d7c809e690 2024-04-26T22:49:36.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fcec3fff640 1 -- 172.21.15.1:0/4153503249 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fced8041040 con 0x55d7c809e690 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcedc010df0 con 0x55d7c809e690 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fcec3fff640 1 -- 172.21.15.1:0/4153503249 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fced8004430 con 0x55d7c809e690 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 10 client.15276 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d7c8120920 con 0x55d7c809e690 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fcec3fff640 1 -- 172.21.15.1:0/4153503249 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcedc010df0 con 0x55d7c809e690 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fcec3fff640 1 -- 172.21.15.1:0/4153503249 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fced804fce0 con 0x55d7c809e690 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 10 client.15276 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 10 client.15276 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 20 client.15276 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c81251a0 0x55d7c81215e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 4 client.15276 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 --> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] -- command(tid 0: {"prefix": "status"}) -- 0x55d7c800b1c0 con 0x55d7c81251a0 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.566+0000 7fcee3fff640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c81251a0 0x55d7c81215e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fcec3fff640 10 client.15276 ms_handle_connect on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fcee3fff640 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c81251a0 0x55d7c81215e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4802908e20b7c39 server_cookie=e98de6539fe51057 in_seq=0 out_seq=0 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fcec3fff640 1 -- 172.21.15.1:0/4153503249 <== mds.3 v2:172.21.15.38:6836/1812912662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d7c800b1c0 con 0x55d7c81251a0 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fcec3fff640 10 client.15276 handle_command_reply: tid=0 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 client.15276 shutdown 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 msgr2=0x55d7c8117ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 0x55d7c8117ab0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fced8005c40 tx=0x7fced80314a0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 shutdown_connections 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7c80a41f0 0x55d7c811f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:6836/1812912662,v1:172.21.15.38:6837/1812912662] conn(0x55d7c81251a0 0x55d7c81215e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7c809f080 0x55d7c8117ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 --2- 172.21.15.1:0/4153503249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d7c809e690 0x55d7c8117ab0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fcec3fff640 0 client.15276 ms_handle_reset on v2:172.21.15.38:6836/1812912662 2024-04-26T22:49:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 >> 172.21.15.1:0/4153503249 conn(0x55d7c8099d00 msgr2=0x55d7c80a7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 shutdown_connections 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 1 -- 172.21.15.1:0/4153503249 wait complete. 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.567+0000 7fceed7b3740 20 client.15276 trim_cache size 0 max 16384 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 3, 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24473, 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:36.576 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:36.577 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 110.74434525300001, 2024-04-26T22:49:36.577 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:36.577 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:36.577 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:36.577 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 132.28453682700001 2024-04-26T22:49:36.577 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:36.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fced40a3c40 msgr2=0x7fced40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fced40a3c40 0x7fced40a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55d7c809f8e0 tx=0x7fcedc0098e0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 msgr2=0x7fcee4198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 0x7fcee4198040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fced000e9f0 tx=0x7fced000eeb0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 shutdown_connections 2024-04-26T22:49:36.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fced40a3c40 0x7fced40a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee4103420 0x7fcee419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcee4102a60 0x7fcee4198040 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 --2- 172.21.15.1:0/741865441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcee4101870 0x7fcee4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 >> 172.21.15.1:0/741865441 conn(0x7fcee40fce00 msgr2=0x7fcee4105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 shutdown_connections 2024-04-26T22:49:36.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.568+0000 7fceeaa93640 1 -- 172.21.15.1:0/741865441 wait complete. 2024-04-26T22:49:36.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-26T22:49:36.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/1581416521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 msgr2=0x7fbc64104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/1581416521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64104000 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009900 tx=0x7fbc4c02a410 comp rx=0 tx=0).stop 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/1581416521 shutdown_connections 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/1581416521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 0x7fbc6410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/1581416521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64104000 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/1581416521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 0x7fbc64102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/1581416521 >> 172.21.15.1:0/1581416521 conn(0x7fbc640fdf00 msgr2=0x7fbc64100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/1581416521 shutdown_connections 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.684+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/1581416521 wait complete. 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 Processor -- start 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 -- start start 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 0x7fbc64198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 0x7fbc641a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6410e4c0 con 0x7fbc64103b80 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6410e340 con 0x7fbc64104540 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc6b1c6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6410e640 con 0x7fbc64102990 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43080/0 (socket says 172.21.15.1:43080) 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 -- 172.21.15.1:0/780702407 learned_addr learned my addr 172.21.15.1:0/780702407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc68f62640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 0x7fbc64198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc69763640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 0x7fbc641a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 -- 172.21.15.1:0/780702407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 msgr2=0x7fbc64198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 0x7fbc64198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 -- 172.21.15.1:0/780702407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 msgr2=0x7fbc641a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 0x7fbc641a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 -- 172.21.15.1:0/780702407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc641a0810 con 0x7fbc64103b80 2024-04-26T22:49:36.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc69763640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 0x7fbc641a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc68f62640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 0x7fbc64198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.685+0000 7fbc63fff640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64199130 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c005940 tx=0x7fbc4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.686+0000 7fbc61ffb640 1 -- 172.21.15.1:0/780702407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c03f030 con 0x7fbc64103b80 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.686+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc641a0aa0 con 0x7fbc64103b80 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.686+0000 7fbc61ffb640 1 -- 172.21.15.1:0/780702407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4c003cb0 con 0x7fbc64103b80 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.686+0000 7fbc61ffb640 1 -- 172.21.15.1:0/780702407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c035070 con 0x7fbc64103b80 2024-04-26T22:49:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.686+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc641a0fe0 con 0x7fbc64103b80 2024-04-26T22:49:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc61ffb640 1 -- 172.21.15.1:0/780702407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbc4c042030 con 0x7fbc64103b80 2024-04-26T22:49:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 Processor -- start 2024-04-26T22:49:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 -- start start 2024-04-26T22:49:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e041690 0x555e4e041ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 0x555e4e046b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e049680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555e4dfa6740 con 0x555e4e042080 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555e4df423a0 con 0x555e4e0471f0 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555e4dfa2700 con 0x555e4e041690 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.688+0000 7fbc68f62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 0x555e4e046b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 0x555e4e046b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43088/0 (socket says 172.21.15.1:43088) 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 -- 172.21.15.1:0/4092503609 learned_addr learned my addr 172.21.15.1:0/4092503609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc61ffb640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc500a3c40 0x7fbc500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc63fff640 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e049680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc69763640 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e041690 0x555e4e041ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 -- 172.21.15.1:0/4092503609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e041690 msgr2=0x555e4e041ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e041690 0x555e4e041ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 -- 172.21.15.1:0/4092503609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 msgr2=0x555e4e049680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e049680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 -- 172.21.15.1:0/4092503609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e4df5e750 con 0x555e4e042080 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc69763640 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e041690 0x555e4e041ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc69763640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc500a3c40 0x7fbc500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc61ffb640 1 -- 172.21.15.1:0/780702407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fbc4c0f0a70 con 0x7fbc64103b80 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc68f62640 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 0x555e4e046b80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009cf0 tx=0x7fbc540117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c1eaf7ad4b2fc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc5affd640 1 -- 172.21.15.1:0/4092503609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54012450 con 0x555e4e042080 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc69763640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc500a3c40 0x7fbc500a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x555e4e0428e0 tx=0x7fbc5c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc5affd640 1 -- 172.21.15.1:0/4092503609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc54012aa0 con 0x555e4e042080 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.689+0000 7fbc5affd640 1 -- 172.21.15.1:0/4092503609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5400fb90 con 0x555e4e042080 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4092503609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 msgr2=0x555e4e046b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 0x555e4e046b80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009cf0 tx=0x7fbc540117b0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4092503609 shutdown_connections 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e049680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e042080 0x555e4e046b80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4092503609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e041690 0x555e4e041ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4092503609 >> 172.21.15.1:0/4092503609 conn(0x555e4e03cd00 msgr2=0x555e4e03d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4092503609 shutdown_connections 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4092503609 wait complete. 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 -- start start 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.690+0000 7fbc6dfc1740 1 Processor -- start 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e041690 0x555e4e04bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 0x555e4e053060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555e4e04a1a0 con 0x555e4e0471f0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555e4df2dfe0 con 0x555e4e041690 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555e4e04a320 con 0x555e4e042080 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc6dfc1740 10 client.0 _ll_register_callbacks cb 0x555e4df44150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 0x555e4e053060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 0x555e4e053060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43104/0 (socket says 172.21.15.1:43104) 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 -- 172.21.15.1:0/165933911 learned_addr learned my addr 172.21.15.1:0/165933911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc63fff640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc68f62640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e041690 0x555e4e04bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 -- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 msgr2=0x555e4e04c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 -- 172.21.15.1:0/165933911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e041690 msgr2=0x555e4e04bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e041690 0x555e4e04bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 -- 172.21.15.1:0/165933911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4dcb82d0 con 0x555e4e0471f0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc69763640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 0x555e4e053060 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009fd0 tx=0x7fbc5c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfb590fdf3d4ac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.691+0000 7fbc43fff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc43fff640 1 -- 172.21.15.1:0/165933911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c01c920 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc43fff640 1 -- 172.21.15.1:0/165933911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fbc5c01cac0 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc54012aa0 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc43fff640 1 -- 172.21.15.1:0/165933911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbc5c02f070 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 10 client.15288 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555e4dd826b0 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc43fff640 1 -- 172.21.15.1:0/165933911 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc54012aa0 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc43fff640 1 -- 172.21.15.1:0/165933911 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fbc5c01a390 con 0x555e4e0471f0 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 10 client.15288 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 10 client.15288 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 20 client.15288 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e058630 0x555e4e059aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 4 client.15288 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.692+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 --> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e4dfb4350 con 0x555e4e058630 2024-04-26T22:49:36.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.693+0000 7fbc68f62640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e058630 0x555e4e059aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.693+0000 7fbc68f62640 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e058630 0x555e4e059aa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9a5908e02d49dc43 server_cookie=2b632c8fca78ffc8 in_seq=0 out_seq=0 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.693+0000 7fbc43fff640 10 client.15288 ms_handle_connect on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc43fff640 1 -- 172.21.15.1:0/165933911 <== mds.2 v2:172.21.15.186:6834/1379199020 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e4dfb4350 con 0x555e4e058630 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc43fff640 10 client.15288 handle_command_reply: tid=0 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 client.15288 shutdown 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 msgr2=0x555e4e053060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 0x555e4e053060 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009fd0 tx=0x7fbc5c009500 comp rx=0 tx=0).stop 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 shutdown_connections 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e058630 0x555e4e059aa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e0471f0 0x555e4e053060 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc43fff640 0 client.15288 ms_handle_reset on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/165933911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e041690 0x555e4e04bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 >> 172.21.15.1:0/165933911 conn(0x555e4e03cd00 msgr2=0x555e4e03bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 shutdown_connections 2024-04-26T22:49:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/165933911 wait complete. 2024-04-26T22:49:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.694+0000 7fbc6dfc1740 20 client.15288 trim_cache size 0 max 16384 2024-04-26T22:49:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 Processor -- start 2024-04-26T22:49:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 -- start start 2024-04-26T22:49:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e0434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e0439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555e4e057de0 con 0x555e4e041690 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555e4e0584b0 con 0x555e4e0471f0 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.712+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555e4e058b80 con 0x555e4e042080 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e0434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e0434a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43110/0 (socket says 172.21.15.1:43110) 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 -- 172.21.15.1:0/3711246340 learned_addr learned my addr 172.21.15.1:0/3711246340 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc63fff640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e0439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 -- 172.21.15.1:0/3711246340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 msgr2=0x555e4e0439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e0439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc69763640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 -- 172.21.15.1:0/3711246340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 msgr2=0x555e4e052f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 -- 172.21.15.1:0/3711246340 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e4e042910 con 0x555e4e041690 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc63fff640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e0439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc69763640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc68f62640 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e0434a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009980 tx=0x7fbc54005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3fca158cdccfca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc5a7fc640 1 -- 172.21.15.1:0/3711246340 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5402e070 con 0x555e4e041690 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc5a7fc640 1 -- 172.21.15.1:0/3711246340 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc5400f540 con 0x555e4e041690 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.713+0000 7fbc5a7fc640 1 -- 172.21.15.1:0/3711246340 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5402a150 con 0x555e4e041690 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/3711246340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 msgr2=0x555e4e0434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e0434a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009980 tx=0x7fbc54005810 comp rx=0 tx=0).stop 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/3711246340 shutdown_connections 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e0439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/3711246340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e0434a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/3711246340 >> 172.21.15.1:0/3711246340 conn(0x555e4e03cd00 msgr2=0x555e4e03d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/3711246340 shutdown_connections 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/3711246340 wait complete. 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 -- start start 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.714+0000 7fbc6dfc1740 1 Processor -- start 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e04f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555e4e059920 con 0x555e4e041690 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555e4e059250 con 0x555e4e0471f0 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555e4e059ff0 con 0x555e4e042080 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 10 client.0 _ll_register_callbacks cb 0x7fbc5c00f9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e04f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e04f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43120/0 (socket says 172.21.15.1:43120) 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 -- 172.21.15.1:0/4181592830 learned_addr learned my addr 172.21.15.1:0/4181592830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc63fff640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 -- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 msgr2=0x555e4e04f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 -- 172.21.15.1:0/4181592830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 msgr2=0x555e4e052570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 -- 172.21.15.1:0/4181592830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4e052b60 con 0x555e4e041690 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc63fff640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc68f62640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e04f0d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009730 tx=0x7fbc5400f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ccba8de8a4b1831 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc43fff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc43fff640 1 -- 172.21.15.1:0/4181592830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5402e040 con 0x555e4e041690 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc43fff640 1 -- 172.21.15.1:0/4181592830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fbc54005070 con 0x555e4e041690 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.715+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc5400f540 con 0x555e4e041690 2024-04-26T22:49:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc43fff640 1 -- 172.21.15.1:0/4181592830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbc540052b0 con 0x555e4e041690 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 10 client.15294 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555e4e0bedf0 con 0x555e4e041690 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc43fff640 1 -- 172.21.15.1:0/4181592830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc5400f540 con 0x555e4e041690 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc43fff640 1 -- 172.21.15.1:0/4181592830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fbc540037f0 con 0x555e4e041690 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 10 client.15294 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 10 client.15294 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 20 client.15294 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e0c33e0 0x555e4e0c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 4 client.15294 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 --> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] -- command(tid 0: {"prefix": "status"}) -- 0x555e4dfae1c0 con 0x555e4e0c33e0 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.716+0000 7fbc63fff640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e0c33e0 0x555e4e0c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc43fff640 10 client.15294 ms_handle_connect on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc63fff640 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e0c33e0 0x555e4e0c20f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d83ef4a8b96ec9c server_cookie=95160044da81c76a in_seq=0 out_seq=0 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc43fff640 1 -- 172.21.15.1:0/4181592830 <== mds.2 v2:172.21.15.186:6834/1379199020 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x555e4dfae1c0 con 0x555e4e0c33e0 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc43fff640 10 client.15294 handle_command_reply: tid=0 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 client.15294 shutdown 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 msgr2=0x555e4e04f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e04f0d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009730 tx=0x7fbc5400f6f0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 shutdown_connections 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:6834/1379199020,v1:172.21.15.186:6835/1379199020] conn(0x555e4e0c33e0 0x555e4e0c20f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555e4e042080 0x555e4e04f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc43fff640 0 client.15294 ms_handle_reset on v2:172.21.15.186:6834/1379199020 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e4e0471f0 0x555e4e052570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 --2- 172.21.15.1:0/4181592830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e4e041690 0x555e4e04f0d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 >> 172.21.15.1:0/4181592830 conn(0x555e4e03cd00 msgr2=0x555e4e0434a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 shutdown_connections 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 1 -- 172.21.15.1:0/4181592830 wait complete. 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.717+0000 7fbc6dfc1740 20 client.15294 trim_cache size 0 max 16384 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 2, 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24475, 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 111.905335608, 2024-04-26T22:49:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 138.397475329 2024-04-26T22:49:36.727 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc500a3c40 msgr2=0x7fbc500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc500a3c40 0x7fbc500a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x555e4e0428e0 tx=0x7fbc5c004a40 comp rx=0 tx=0).stop 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 msgr2=0x7fbc64199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64199130 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c005940 tx=0x7fbc4c0042f0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 shutdown_connections 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbc500a3c40 0x7fbc500a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc64104540 0x7fbc641a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc64103b80 0x7fbc64199130 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 --2- 172.21.15.1:0/780702407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc64102990 0x7fbc64198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 >> 172.21.15.1:0/780702407 conn(0x7fbc640fdf00 msgr2=0x7fbc64106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 shutdown_connections 2024-04-26T22:49:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.718+0000 7fbc6b1c6640 1 -- 172.21.15.1:0/780702407 wait complete. 2024-04-26T22:49:36.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T22:49:36.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 -- 172.21.15.1:0/1143858613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0101880 msgr2=0x7f5df0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 --2- 172.21.15.1:0/1143858613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0101880 0x7f5df0101ca0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009900 tx=0x7f5de002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 -- 172.21.15.1:0/1143858613 shutdown_connections 2024-04-26T22:49:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 --2- 172.21.15.1:0/1143858613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5df0102a70 0x7f5df0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 --2- 172.21.15.1:0/1143858613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0103430 0x7f5df0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 --2- 172.21.15.1:0/1143858613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0101880 0x7f5df0101ca0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 -- 172.21.15.1:0/1143858613 >> 172.21.15.1:0/1143858613 conn(0x7f5df00fce10 msgr2=0x7f5df00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 -- 172.21.15.1:0/1143858613 shutdown_connections 2024-04-26T22:49:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.832+0000 7f5df610a640 1 -- 172.21.15.1:0/1143858613 wait complete. 2024-04-26T22:49:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 Processor -- start 2024-04-26T22:49:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 -- start start 2024-04-26T22:49:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0101880 0x7f5df0198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5df0102a70 0x7f5df01992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 0x7f5df01a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5df010d480 con 0x7f5df0103430 2024-04-26T22:49:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5df010d300 con 0x7f5df0101880 2024-04-26T22:49:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5df610a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5df010d600 con 0x7f5df0102a70 2024-04-26T22:49:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 0x7f5df01a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 0x7f5df01a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43134/0 (socket says 172.21.15.1:43134) 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 -- 172.21.15.1:0/141354901 learned_addr learned my addr 172.21.15.1:0/141354901 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5def7fe640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0101880 0x7f5df0198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5de7fff640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5df0102a70 0x7f5df01992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 -- 172.21.15.1:0/141354901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5df0102a70 msgr2=0x7f5df01992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5df0102a70 0x7f5df01992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 -- 172.21.15.1:0/141354901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0101880 msgr2=0x7f5df0198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0101880 0x7f5df0198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.833+0000 7f5deffff640 1 -- 172.21.15.1:0/141354901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df01a09b0 con 0x7f5df0103430 2024-04-26T22:49:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.834+0000 7f5deffff640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 0x7f5df01a0310 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5de800b970 tx=0x7f5de800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.834+0000 7f5ded7fa640 1 -- 172.21.15.1:0/141354901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de8019070 con 0x7f5df0103430 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.834+0000 7f5ded7fa640 1 -- 172.21.15.1:0/141354901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de8014440 con 0x7f5df0103430 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.834+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df01a0ca0 con 0x7f5df0103430 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.834+0000 7f5ded7fa640 1 -- 172.21.15.1:0/141354901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de80075d0 con 0x7f5df0103430 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.834+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df01a11a0 con 0x7f5df0103430 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.836+0000 7f5ded7fa640 1 -- 172.21.15.1:0/141354901 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5de801c070 con 0x7f5df0103430 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.836+0000 7f5df8e29740 1 Processor -- start 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.836+0000 7f5df8e29740 1 -- start start 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5df8e29740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558e2b60f740 con 0x558e2b6ab080 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5df8e29740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558e2b5ab3a0 con 0x558e2b6aa690 2024-04-26T22:49:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5df8e29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558e2b60b700 con 0x558e2b6b01f0 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43136/0 (socket says 172.21.15.1:43136) 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 -- 172.21.15.1:0/2602785606 learned_addr learned my addr 172.21.15.1:0/2602785606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5ded7fa640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5ddc0a3cc0 0x7f5ddc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5de7fff640 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 -- 172.21.15.1:0/2602785606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 msgr2=0x558e2b6b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 -- 172.21.15.1:0/2602785606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 msgr2=0x558e2b6aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6aaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 -- 172.21.15.1:0/2602785606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2b5c7750 con 0x558e2b6ab080 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5deffff640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5ddc0a3cc0 0x7f5ddc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5de7fff640 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5ded7fa640 1 -- 172.21.15.1:0/141354901 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f5de80cfa60 con 0x7f5df0103430 2024-04-26T22:49:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5def7fe640 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6afb80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5df01028d0 tx=0x7f5de0031de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e560ce07576ce071 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5de5ffb640 1 -- 172.21.15.1:0/2602785606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0035030 con 0x558e2b6ab080 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5de5ffb640 1 -- 172.21.15.1:0/2602785606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de0040070 con 0x558e2b6ab080 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.837+0000 7f5de5ffb640 1 -- 172.21.15.1:0/2602785606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0003b80 con 0x558e2b6ab080 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 -- 172.21.15.1:0/2602785606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 msgr2=0x558e2b6afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5deffff640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5ddc0a3cc0 0x7f5ddc0a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5de806e7f0 tx=0x7f5de806e820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6afb80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5df01028d0 tx=0x7f5de0031de0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 -- 172.21.15.1:0/2602785606 shutdown_connections 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6afb80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2602785606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 -- 172.21.15.1:0/2602785606 >> 172.21.15.1:0/2602785606 conn(0x558e2b6a5d00 msgr2=0x558e2b6a6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 -- 172.21.15.1:0/2602785606 shutdown_connections 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 -- 172.21.15.1:0/2602785606 wait complete. 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.838+0000 7f5df8e29740 1 -- start start 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 Processor -- start 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6ab080 0x558e2b6b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 0x558e2b6bc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6b3140 con 0x558e2b6b01f0 2024-04-26T22:49:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558e2b596fe0 con 0x558e2b6aa690 2024-04-26T22:49:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6b32c0 con 0x558e2b6ab080 2024-04-26T22:49:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5df8e29740 10 client.0 _ll_register_callbacks cb 0x558e2b5ad150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 0x558e2b6bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 0x558e2b6bc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43152/0 (socket says 172.21.15.1:43152) 2024-04-26T22:49:36.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 -- 172.21.15.1:0/1916828617 learned_addr learned my addr 172.21.15.1:0/1916828617 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5def7fe640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6ab080 0x558e2b6b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 -- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6ab080 msgr2=0x558e2b6b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6ab080 0x558e2b6b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 -- 172.21.15.1:0/1916828617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 msgr2=0x558e2b6b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 -- 172.21.15.1:0/1916828617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2b3212d0 con 0x558e2b6b01f0 2024-04-26T22:49:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5de7fff640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 0x558e2b6bc090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80148c0 tx=0x7f5dd800d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a8a8979d08f75b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.839+0000 7f5dceffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5dceffd640 1 -- 172.21.15.1:0/1916828617 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd800faf0 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5dceffd640 1 -- 172.21.15.1:0/1916828617 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f5dd800fc90 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de0040070 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5dceffd640 1 -- 172.21.15.1:0/1916828617 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5dd800fe90 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 10 client.15306 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558e2b3eb6b0 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5dceffd640 1 -- 172.21.15.1:0/1916828617 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de0040070 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5dceffd640 1 -- 172.21.15.1:0/1916828617 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f5dd8015040 con 0x558e2b6b01f0 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 10 client.15306 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 10 client.15306 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 20 client.15306 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b6c14b0 0x558e2b6c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 4 client.15306 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 --> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e2b61d350 con 0x558e2b6c14b0 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.840+0000 7f5def7fe640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b6c14b0 0x558e2b6c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.841+0000 7f5def7fe640 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b6c14b0 0x558e2b6c2920 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21c04f47aa292716 server_cookie=b531ece6ecab76d3 in_seq=0 out_seq=0 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.841+0000 7f5dceffd640 10 client.15306 ms_handle_connect on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5dceffd640 1 -- 172.21.15.1:0/1916828617 <== mds.1 v2:172.21.15.186:6836/786403012 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e2b61d350 con 0x558e2b6c14b0 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5dceffd640 10 client.15306 handle_command_reply: tid=0 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 client.15306 shutdown 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 msgr2=0x558e2b6bc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 0x558e2b6bc090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80148c0 tx=0x7f5dd800d3c0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 shutdown_connections 2024-04-26T22:49:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b6c14b0 0x558e2b6c2920 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6ab080 0x558e2b6b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6b01f0 0x558e2b6bc090 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5dceffd640 0 client.15306 ms_handle_reset on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 --2- 172.21.15.1:0/1916828617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 >> 172.21.15.1:0/1916828617 conn(0x558e2b6a5d00 msgr2=0x558e2b6a4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 shutdown_connections 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 1 -- 172.21.15.1:0/1916828617 wait complete. 2024-04-26T22:49:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.842+0000 7f5df8e29740 20 client.15306 trim_cache size 0 max 16384 2024-04-26T22:49:36.857 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:36 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3803321625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:36.857 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:36 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2759142092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:36.857 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:36 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3820876505' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:36.857 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:36 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3803321625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:36.857 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:36 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2759142092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:36.857 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:36 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3820876505' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 Processor -- start 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 -- start start 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6cac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6c0c60 con 0x558e2b6ab080 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6c1330 con 0x558e2b6aa690 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5df8e29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6c1a00 con 0x558e2b6b01f0 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5def7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5def7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43164/0 (socket says 172.21.15.1:43164) 2024-04-26T22:49:36.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.860+0000 7f5def7fe640 1 -- 172.21.15.1:0/2496386254 learned_addr learned my addr 172.21.15.1:0/2496386254 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5de7fff640 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5def7fe640 1 -- 172.21.15.1:0/2496386254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 msgr2=0x558e2b6bbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5def7fe640 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5def7fe640 1 -- 172.21.15.1:0/2496386254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 msgr2=0x558e2b6cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5def7fe640 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5def7fe640 1 -- 172.21.15.1:0/2496386254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2b6af780 con 0x558e2b6ab080 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5de7fff640 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5def7fe640 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6cb140 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5de0000c00 tx=0x7f5de0047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d88daa8319f76d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5de57fa640 1 -- 172.21.15.1:0/2496386254 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0035040 con 0x558e2b6ab080 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5de57fa640 1 -- 172.21.15.1:0/2496386254 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de0053070 con 0x558e2b6ab080 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5de57fa640 1 -- 172.21.15.1:0/2496386254 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0045cd0 con 0x558e2b6ab080 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5df8e29740 1 -- 172.21.15.1:0/2496386254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 msgr2=0x558e2b6cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.861+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6cb140 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5de0000c00 tx=0x7f5de0047040 comp rx=0 tx=0).stop 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- 172.21.15.1:0/2496386254 shutdown_connections 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6cb140 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 --2- 172.21.15.1:0/2496386254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- 172.21.15.1:0/2496386254 >> 172.21.15.1:0/2496386254 conn(0x558e2b6a5d00 msgr2=0x558e2b6a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- 172.21.15.1:0/2496386254 shutdown_connections 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- 172.21.15.1:0/2496386254 wait complete. 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- start start 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 Processor -- start 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6c27a0 con 0x558e2b6ab080 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6c20d0 con 0x558e2b6aa690 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558e2b6c2e70 con 0x558e2b6b01f0 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5df8e29740 10 client.0 _ll_register_callbacks cb 0x7f5dd800d260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5def7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5def7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6b8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43174/0 (socket says 172.21.15.1:43174) 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.862+0000 7f5def7fe640 1 -- 172.21.15.1:0/4209671371 learned_addr learned my addr 172.21.15.1:0/4209671371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5de7fff640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5de7fff640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5def7fe640 1 -- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 msgr2=0x558e2b6bb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5def7fe640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5def7fe640 1 -- 172.21.15.1:0/4209671371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 msgr2=0x558e2b6b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5def7fe640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5def7fe640 1 -- 172.21.15.1:0/4209671371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2b6bbdb0 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5de7fff640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5def7fe640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6b8a30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5de0031c70 tx=0x7f5de0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e559ba31b63ae433 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5dceffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5dceffd640 1 -- 172.21.15.1:0/4209671371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de005b070 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5dceffd640 1 -- 172.21.15.1:0/4209671371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f5de0045c30 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de0053070 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5dceffd640 1 -- 172.21.15.1:0/4209671371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5de0035040 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5df8e29740 10 client.15312 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558e2b727dd0 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5dceffd640 1 -- 172.21.15.1:0/4209671371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de0053070 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.863+0000 7f5dceffd640 1 -- 172.21.15.1:0/4209671371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f5de003ca90 con 0x558e2b6ab080 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 10 client.15312 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 10 client.15312 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T22:49:36.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 20 client.15312 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b72c310 0x558e2b72b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 4 client.15312 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 --> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] -- command(tid 0: {"prefix": "status"}) -- 0x558e2b6171c0 con 0x558e2b72c310 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5de7fff640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b72c310 0x558e2b72b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5de7fff640 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b72c310 0x558e2b72b020 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e113c6582e3c655 server_cookie=993a331112f6dea1 in_seq=0 out_seq=0 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5dceffd640 10 client.15312 ms_handle_connect on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5dceffd640 1 -- 172.21.15.1:0/4209671371 <== mds.1 v2:172.21.15.186:6836/786403012 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x558e2b6171c0 con 0x558e2b72c310 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5dceffd640 10 client.15312 handle_command_reply: tid=0 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 1 client.15312 shutdown 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 msgr2=0x558e2b6b8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.864+0000 7f5df8e29740 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6b8a30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5de0031c70 tx=0x7f5de0031a50 comp rx=0 tx=0).stop 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 shutdown_connections 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:6836/786403012,v1:172.21.15.186:6837/786403012] conn(0x558e2b72c310 0x558e2b72b020 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e2b6b01f0 0x558e2b6bb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558e2b6ab080 0x558e2b6b8a30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5dceffd640 0 client.15312 ms_handle_reset on v2:172.21.15.186:6836/786403012 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 --2- 172.21.15.1:0/4209671371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e2b6aa690 0x558e2b6b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 >> 172.21.15.1:0/4209671371 conn(0x558e2b6a5d00 msgr2=0x558e2b6ac4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 shutdown_connections 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 1 -- 172.21.15.1:0/4209671371 wait complete. 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df8e29740 20 client.15312 trim_cache size 0 max 16384 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-04-26T22:49:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24487, 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 113.057362908, 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 135.677952011 2024-04-26T22:49:36.875 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5ddc0a3cc0 msgr2=0x7f5ddc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5ddc0a3cc0 0x7f5ddc0a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5de806e7f0 tx=0x7f5de806e820 comp rx=0 tx=0).stop 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 msgr2=0x7f5df01a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 0x7f5df01a0310 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5de800b970 tx=0x7f5de800be30 comp rx=0 tx=0).stop 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 shutdown_connections 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5ddc0a3cc0 0x7f5ddc0a6180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5df0102a70 0x7f5df01992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0103430 0x7f5df01a0310 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 --2- 172.21.15.1:0/141354901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0101880 0x7f5df0198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 >> 172.21.15.1:0/141354901 conn(0x7f5df00fce10 msgr2=0x7f5df006b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 shutdown_connections 2024-04-26T22:49:36.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.865+0000 7f5df610a640 1 -- 172.21.15.1:0/141354901 wait complete. 2024-04-26T22:49:36.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T22:49:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 -- 172.21.15.1:0/2022587963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c101880 msgr2=0x7f138c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 --2- 172.21.15.1:0/2022587963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c101880 0x7f138c101ca0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f137c009900 tx=0x7f137c02a410 comp rx=0 tx=0).stop 2024-04-26T22:49:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 -- 172.21.15.1:0/2022587963 shutdown_connections 2024-04-26T22:49:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 --2- 172.21.15.1:0/2022587963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c102a70 0x7f138c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 --2- 172.21.15.1:0/2022587963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 0x7f138c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 --2- 172.21.15.1:0/2022587963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c101880 0x7f138c101ca0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 -- 172.21.15.1:0/2022587963 >> 172.21.15.1:0/2022587963 conn(0x7f138c0fce10 msgr2=0x7f138c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 -- 172.21.15.1:0/2022587963 shutdown_connections 2024-04-26T22:49:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.981+0000 7f1390bb4640 1 -- 172.21.15.1:0/2022587963 wait complete. 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 Processor -- start 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 -- start start 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c101880 0x7f138c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 0x7f138c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 0x7f138c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f138c10d400 con 0x7f138c102a70 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f138c10d280 con 0x7f138c103430 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1390bb4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138c10d580 con 0x7f138c101880 2024-04-26T22:49:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 0x7f138c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 0x7f138c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43196/0 (socket says 172.21.15.1:43196) 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 -- 172.21.15.1:0/1920618597 learned_addr learned my addr 172.21.15.1:0/1920618597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f138a59c640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c101880 0x7f138c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 0x7f138c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 -- 172.21.15.1:0/1920618597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c101880 msgr2=0x7f138c197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c101880 0x7f138c197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 -- 172.21.15.1:0/1920618597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 msgr2=0x7f138c19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 0x7f138c19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 -- 172.21.15.1:0/1920618597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c19f780 con 0x7f138c102a70 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 0x7f138c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.982+0000 7f1389d9b640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 0x7f138c1980f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f138000d990 tx=0x7f138000de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.983+0000 7f137b7fe640 1 -- 172.21.15.1:0/1920618597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138001b070 con 0x7f138c102a70 2024-04-26T22:49:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.983+0000 7f137b7fe640 1 -- 172.21.15.1:0/1920618597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138000edf0 con 0x7f138c102a70 2024-04-26T22:49:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.983+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138c06b060 con 0x7f138c102a70 2024-04-26T22:49:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.983+0000 7f137b7fe640 1 -- 172.21.15.1:0/1920618597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380014800 con 0x7f138c102a70 2024-04-26T22:49:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.983+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138c06b560 con 0x7f138c102a70 2024-04-26T22:49:36.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:36 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3803321625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T22:49:36.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:36 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2759142092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T22:49:36.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:36 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3820876505' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.985+0000 7f137b7fe640 1 -- 172.21.15.1:0/1920618597 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f138001e070 con 0x7f138c102a70 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.985+0000 7f13939ad740 1 Processor -- start 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 -- start start 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1bbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1be680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a70b11b740 con 0x55a70b1b6690 2024-04-26T22:49:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a70b0b73a0 con 0x55a70b1bc1f0 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f13939ad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a70b117700 con 0x55a70b1b7080 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f137b7fe640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13740a3c40 0x7f13740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43210/0 (socket says 172.21.15.1:43210) 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 -- 172.21.15.1:0/1699362900 learned_addr learned my addr 172.21.15.1:0/1699362900 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138a59c640 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1bbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f1389d9b640 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13740a3c40 0x7f13740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f137b7fe640 1 -- 172.21.15.1:0/1920618597 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f13800d1c70 con 0x7f138c102a70 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 -- 172.21.15.1:0/1699362900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 msgr2=0x55a70b1bbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1bbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 -- 172.21.15.1:0/1699362900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 msgr2=0x55a70b1be680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1be680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 -- 172.21.15.1:0/1699362900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a70b0d3750 con 0x55a70b1b6690 2024-04-26T22:49:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.986+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1b6ab0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1384012a80 tx=0x7f1384012f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab697ff888ffaff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f1378ff9640 1 -- 172.21.15.1:0/1699362900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384022070 con 0x55a70b1b6690 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f1378ff9640 1 -- 172.21.15.1:0/1699362900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1384012450 con 0x55a70b1b6690 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f1378ff9640 1 -- 172.21.15.1:0/1699362900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840105e0 con 0x55a70b1b6690 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13740a3c40 0x7f13740a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f138401f000 tx=0x7f138401b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 -- 172.21.15.1:0/1699362900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 msgr2=0x55a70b1b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1b6ab0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1384012a80 tx=0x7f1384012f40 comp rx=0 tx=0).stop 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 -- 172.21.15.1:0/1699362900 shutdown_connections 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1bbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1be680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 --2- 172.21.15.1:0/1699362900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1b6ab0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.987+0000 7f13939ad740 1 -- 172.21.15.1:0/1699362900 >> 172.21.15.1:0/1699362900 conn(0x55a70b1b1d00 msgr2=0x55a70b1b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 -- 172.21.15.1:0/1699362900 shutdown_connections 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 -- 172.21.15.1:0/1699362900 wait complete. 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 -- start start 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 Processor -- start 2024-04-26T22:49:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b6690 0x55a70b1c0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1bf1a0 con 0x55a70b1bc1f0 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a70b0a2fe0 con 0x55a70b1b6690 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1bf320 con 0x55a70b1b7080 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f13939ad740 10 client.0 _ll_register_callbacks cb 0x55a70b0b9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43218/0 (socket says 172.21.15.1:43218) 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 -- 172.21.15.1:0/1574734243 learned_addr learned my addr 172.21.15.1:0/1574734243 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138a59c640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b6690 0x55a70b1c0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f1389d9b640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 -- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 msgr2=0x55a70b1c1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 -- 172.21.15.1:0/1574734243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b6690 msgr2=0x55a70b1c0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b6690 0x55a70b1c0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.988+0000 7f138ad9d640 1 -- 172.21.15.1:0/1574734243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a70ae2d2d0 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1389d9b640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f138ad9d640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c8060 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1384009550 tx=0x7f138401ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6f441b5c057bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1371ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1371ffb640 1 -- 172.21.15.1:0/1574734243 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384022040 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1371ffb640 1 -- 172.21.15.1:0/1574734243 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f1384010590 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1384012450 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1371ffb640 1 -- 172.21.15.1:0/1574734243 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1384004b40 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 10 client.15324 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a70aef76b0 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1371ffb640 1 -- 172.21.15.1:0/1574734243 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1384012450 con 0x55a70b1bc1f0 2024-04-26T22:49:36.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f1371ffb640 1 -- 172.21.15.1:0/1574734243 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f138400f620 con 0x55a70b1bc1f0 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 10 client.15324 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 10 client.15324 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 20 client.15324 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b1cd3b0 0x55a70b1ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 4 client.15324 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.989+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 --> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a70b129350 con 0x55a70b1cd3b0 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.990+0000 7f138a59c640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b1cd3b0 0x55a70b1ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.990+0000 7f138a59c640 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b1cd3b0 0x55a70b1ce820 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9713fbbce59ae225 server_cookie=b4ae417c65e46a03 in_seq=0 out_seq=0 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.990+0000 7f1371ffb640 10 client.15324 ms_handle_connect on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f1371ffb640 1 -- 172.21.15.1:0/1574734243 <== mds.0 v2:172.21.15.186:6838/3709434074 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a70b129350 con 0x55a70b1cd3b0 2024-04-26T22:49:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f1371ffb640 10 client.15324 handle_command_reply: tid=0 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 client.15324 shutdown 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 msgr2=0x55a70b1c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c8060 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1384009550 tx=0x7f138401ef50 comp rx=0 tx=0).stop 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 shutdown_connections 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b1cd3b0 0x55a70b1ce820 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c8060 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f1371ffb640 0 client.15324 ms_handle_reset on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 --2- 172.21.15.1:0/1574734243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b6690 0x55a70b1c0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 >> 172.21.15.1:0/1574734243 conn(0x55a70b1b1d00 msgr2=0x55a70b1b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 shutdown_connections 2024-04-26T22:49:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 1 -- 172.21.15.1:0/1574734243 wait complete. 2024-04-26T22:49:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:36.991+0000 7f13939ad740 20 client.15324 trim_cache size 0 max 16384 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.009+0000 7f13939ad740 1 Processor -- start 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.009+0000 7f13939ad740 1 -- start start 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f13939ad740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f13939ad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f13939ad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f13939ad740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1ccb60 con 0x55a70b1b6690 2024-04-26T22:49:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f13939ad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1cd230 con 0x55a70b1bc1f0 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f13939ad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1cd900 con 0x55a70b1b7080 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1be5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43232/0 (socket says 172.21.15.1:43232) 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 -- 172.21.15.1:0/3215885817 learned_addr learned my addr 172.21.15.1:0/3215885817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 -- 172.21.15.1:0/3215885817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 msgr2=0x55a70b1c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 -- 172.21.15.1:0/3215885817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 msgr2=0x55a70b1c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138ad9d640 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 -- 172.21.15.1:0/3215885817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a70b1b7460 con 0x55a70b1b6690 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138ad9d640 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f138a59c640 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1be5e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f137c005ab0 tx=0x7f137c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a196119b00661abe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f1373fff640 1 -- 172.21.15.1:0/3215885817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c035040 con 0x55a70b1b6690 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f1373fff640 1 -- 172.21.15.1:0/3215885817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f137c04a070 con 0x55a70b1b6690 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.010+0000 7f1373fff640 1 -- 172.21.15.1:0/3215885817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c005430 con 0x55a70b1b6690 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 -- 172.21.15.1:0/3215885817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 msgr2=0x55a70b1be5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1be5e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f137c005ab0 tx=0x7f137c005a20 comp rx=0 tx=0).stop 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 -- 172.21.15.1:0/3215885817 shutdown_connections 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b7080 0x55a70b1c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 --2- 172.21.15.1:0/3215885817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1b6690 0x55a70b1be5e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 -- 172.21.15.1:0/3215885817 >> 172.21.15.1:0/3215885817 conn(0x55a70b1b1d00 msgr2=0x55a70b1b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 -- 172.21.15.1:0/3215885817 shutdown_connections 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 -- 172.21.15.1:0/3215885817 wait complete. 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 -- start start 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 Processor -- start 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b6690 0x55a70b1c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.011+0000 7f13939ad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b7080 0x55a70b1c3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1ce6a0 con 0x55a70b1bc1f0 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1cdfd0 con 0x55a70b1b7080 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a70b1ced70 con 0x55a70b1b6690 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 10 client.0 _ll_register_callbacks cb 0x7f1384020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T22:49:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43248/0 (socket says 172.21.15.1:43248) 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 -- 172.21.15.1:0/731582068 learned_addr learned my addr 172.21.15.1:0/731582068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138a59c640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b6690 0x55a70b1c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 -- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b6690 msgr2=0x55a70b1c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b6690 0x55a70b1c36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 -- 172.21.15.1:0/731582068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b7080 msgr2=0x55a70b1c3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b7080 0x55a70b1c3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 -- 172.21.15.1:0/731582068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a70b1c7140 con 0x55a70b1bc1f0 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138a59c640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b6690 0x55a70b1c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f138ad9d640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c6b50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1384002230 tx=0x7f1384009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd1044a67a02efb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:37.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f1371ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f1371ffb640 1 -- 172.21.15.1:0/731582068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384022040 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f1371ffb640 1 -- 172.21.15.1:0/731582068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f138400f3e0 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f137c04a070 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f1371ffb640 1 -- 172.21.15.1:0/731582068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f138404c050 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 10 client.15330 fetch_fsmap learned FSMap version 25 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.012+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a70b233360 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f1371ffb640 1 -- 172.21.15.1:0/731582068 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f137c04a070 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f1371ffb640 1 -- 172.21.15.1:0/731582068 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f13840328d0 con 0x55a70b1bc1f0 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f13939ad740 10 client.15330 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f13939ad740 10 client.15330 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f13939ad740 20 client.15330 populate_metadata read hostname 'smithi001' 2024-04-26T22:49:37.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f13939ad740 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b237e00 0x55a70b236b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f13939ad740 4 client.15330 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 --> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] -- command(tid 0: {"prefix": "status"}) -- 0x55a70b1231c0 con 0x55a70b237e00 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f138a59c640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b237e00 0x55a70b236b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f1371ffb640 10 client.15330 ms_handle_connect on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.013+0000 7f138a59c640 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b237e00 0x55a70b236b10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=becd2fba26bf0c51 server_cookie=5d2b5f48a04d4c45 in_seq=0 out_seq=0 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1371ffb640 1 -- 172.21.15.1:0/731582068 <== mds.0 v2:172.21.15.186:6838/3709434074 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55a70b1231c0 con 0x55a70b237e00 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1371ffb640 10 client.15330 handle_command_reply: tid=0 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 client.15330 shutdown 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 msgr2=0x55a70b1c6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c6b50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1384002230 tx=0x7f1384009e10 comp rx=0 tx=0).stop 2024-04-26T22:49:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 shutdown_connections 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:6838/3709434074,v1:172.21.15.186:6839/3709434074] conn(0x55a70b237e00 0x55a70b236b10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a70b1bc1f0 0x55a70b1c6b50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a70b1b7080 0x55a70b1c3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1371ffb640 0 client.15330 ms_handle_reset on v2:172.21.15.186:6838/3709434074 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 --2- 172.21.15.1:0/731582068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a70b1b6690 0x55a70b1c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 >> 172.21.15.1:0/731582068 conn(0x55a70b1b1d00 msgr2=0x55a70b1c0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 shutdown_connections 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 1 -- 172.21.15.1:0/731582068 wait complete. 2024-04-26T22:49:37.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f13939ad740 20 client.15330 trim_cache size 0 max 16384 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "bb8b501a-041d-11ef-bc93-c7b262605968", 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24514, 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 115.2479017, 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 87, 2024-04-26T22:49:37.024 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T22:49:37.025 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 129.86002775700001 2024-04-26T22:49:37.025 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T22:49:37.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13740a3c40 msgr2=0x7f13740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:37.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13740a3c40 0x7f13740a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f138401f000 tx=0x7f138401b000 comp rx=0 tx=0).stop 2024-04-26T22:49:37.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 msgr2=0x7f138c1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 0x7f138c1980f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f138000d990 tx=0x7f138000de50 comp rx=0 tx=0).stop 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 shutdown_connections 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f13740a3c40 0x7f13740a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f138c103430 0x7f138c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138c102a70 0x7f138c1980f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 --2- 172.21.15.1:0/1920618597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f138c101880 0x7f138c197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.014+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 >> 172.21.15.1:0/1920618597 conn(0x7f138c0fce10 msgr2=0x7f138c105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.015+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 shutdown_connections 2024-04-26T22:49:37.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:37.015+0000 7f1390bb4640 1 -- 172.21.15.1:0/1920618597 wait complete. 2024-04-26T22:49:37.027 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T22:49:37.037 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T22:49:37.037 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T22:49:37.037 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T22:49:37.037 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T22:49:37.132 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T22:49:37.132 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T22:49:37.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:37.133 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T22:49:37.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:37.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-04-26T22:49:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:Cannot find device "ceph-brx" 2024-04-26T22:49:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:49:37.217 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-26T22:49:37.217 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T22:49:37.217 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T22:49:37.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid bb8b501a-041d-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T22:49:37.501 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T22:49:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:37 smithi001 ceph-mon[29435]: pgmap v350: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:37 smithi038 ceph-mon[32433]: pgmap v350: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:37 smithi186 ceph-mon[32774]: pgmap v350: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:39.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 -- 172.21.15.1:0/2536903032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 msgr2=0x7f023c1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 --2- 172.21.15.1:0/2536903032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c1080c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f022c0099c0 tx=0x7f022c009cd0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 -- 172.21.15.1:0/2536903032 shutdown_connections 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 --2- 172.21.15.1:0/2536903032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f023c06bd50 0x7f023c06c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 --2- 172.21.15.1:0/2536903032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f023c1087f0 0x7f023c110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 --2- 172.21.15.1:0/2536903032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c1080c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 -- 172.21.15.1:0/2536903032 >> 172.21.15.1:0/2536903032 conn(0x7f023c0792d0 msgr2=0x7f023c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 -- 172.21.15.1:0/2536903032 shutdown_connections 2024-04-26T22:49:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.336+0000 7f024381e700 1 -- 172.21.15.1:0/2536903032 wait complete. 2024-04-26T22:49:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 Processor -- start 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 -- start start 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f023c06bd50 0x7f023c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f023c1087f0 0x7f023c1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f023c1136b0 con 0x7f023c06c700 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f023c113530 con 0x7f023c1087f0 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f024381e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f023c113830 con 0x7f023c06bd50 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f0240db9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f0240db9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c19fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43270/0 (socket says 172.21.15.1:43270) 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f0240db9700 1 -- 172.21.15.1:0/1299443909 learned_addr learned my addr 172.21.15.1:0/1299443909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f02415ba700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f023c06bd50 0x7f023c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.337+0000 7f0241dbb700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f023c1087f0 0x7f023c1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f0240db9700 1 -- 172.21.15.1:0/1299443909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f023c06bd50 msgr2=0x7f023c19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f0240db9700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f023c06bd50 0x7f023c19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f0240db9700 1 -- 172.21.15.1:0/1299443909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f023c1087f0 msgr2=0x7f023c1999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f0240db9700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f023c1087f0 0x7f023c1999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f0240db9700 1 -- 172.21.15.1:0/1299443909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c19a150 con 0x7f023c06c700 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f0240db9700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c19fd20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f022c00b580 tx=0x7f022c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c027070 con 0x7f023c06c700 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f022c0040a0 con 0x7f023c06c700 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c19a3e0 con 0x7f023c06c700 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c00f460 con 0x7f023c06c700 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.338+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c1a66b0 con 0x7f023c06c700 2024-04-26T22:49:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.339+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023c104b70 con 0x7f023c06c700 2024-04-26T22:49:39.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.340+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f022c02a070 con 0x7f023c06c700 2024-04-26T22:49:39.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.341+0000 7f02327fc700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f02280a3d50 0x7f02280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:39.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.341+0000 7f02415ba700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f02280a3d50 0x7f02280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.341+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f022c0d1570 con 0x7f023c06c700 2024-04-26T22:49:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.341+0000 7f02415ba700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f02280a3d50 0x7f02280a6210 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f023c1082a0 tx=0x7f0238009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:39.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.346+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f022c05f060 con 0x7f023c06c700 2024-04-26T22:49:39.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.604+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f023c19aed0 con 0x7f023c06c700 2024-04-26T22:49:39.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.605+0000 7f02327fc700 1 -- 172.21.15.1:0/1299443909 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f022c013070 con 0x7f023c06c700 2024-04-26T22:49:39.605 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T22:49:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f02280a3d50 msgr2=0x7f02280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f02280a3d50 0x7f02280a6210 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f023c1082a0 tx=0x7f0238009190 comp rx=0 tx=0).stop 2024-04-26T22:49:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 msgr2=0x7f023c19fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c19fd20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f022c00b580 tx=0x7f022c0047b0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 shutdown_connections 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f02280a3d50 0x7f02280a6210 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f023c06bd50 0x7f023c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f023c1087f0 0x7f023c1999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 --2- 172.21.15.1:0/1299443909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f023c06c700 0x7f023c19fd20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 >> 172.21.15.1:0/1299443909 conn(0x7f023c0792d0 msgr2=0x7f023c100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 shutdown_connections 2024-04-26T22:49:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:39.607+0000 7f024381e700 1 -- 172.21.15.1:0/1299443909 wait complete. 2024-04-26T22:49:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:39 smithi001 ceph-mon[29435]: pgmap v351: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:39 smithi038 ceph-mon[32433]: pgmap v351: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:39 smithi186 ceph-mon[32774]: pgmap v351: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T22:49:40.270 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T22:49:40.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:40.271 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-04-26T22:49:40.287 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T22:49:40.287 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T22:49:40.287 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T22:49:40.287 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-26T22:49:40.288 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42111sec preferred_lft 42111sec 2024-04-26T22:49:40.289 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-04-26T22:49:40.289 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T22:49:40.289 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T22:49:40.289 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-04-26T22:49:40.289 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-04-26T22:49:40.289 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T22:49:40.289 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T22:49:40.289 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T22:49:40.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-04-26T22:49:40.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-04-26T22:49:40.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-04-26T22:49:40.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T22:49:40.290 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T22:49:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T22:49:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:40.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:40.477 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T22:49:40.503 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-04-26T22:49:40.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:40.505 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-04-26T22:49:40.576 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-04-26T22:49:40.576 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T22:49:40.576 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T22:49:40.577 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T22:49:40.577 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T22:49:40.577 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T22:49:40.577 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T22:49:40.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T22:49:40.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T22:49:40.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T22:49:40.577 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T22:49:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T22:49:40.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:40 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1299443909' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T22:49:40.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:40 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1299443909' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T22:49:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:40.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:40.878 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T22:49:40.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:40.894 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-04-26T22:49:40.950 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T22:49:40.950 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T22:49:40.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T22:49:40.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T22:49:40.950 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T22:49:40.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:40 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1299443909' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T22:49:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T22:49:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.071 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T22:49:41.071 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T22:49:41.071 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T22:49:41.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T22:49:41.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T22:49:41.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T22:49:41.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T22:49:41.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T22:49:41.072 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T22:49:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T22:49:41.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.198 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T22:49:41.198 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T22:49:41.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-04-26T22:49:41.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T22:49:41.198 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T22:49:41.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T22:49:41.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.276 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T22:49:41.276 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T22:49:41.291 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T22:49:41.291 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T22:49:41.291 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T22:49:41.346 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T22:49:41.346 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T22:49:41.346 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/69154049-25b2-4b6a-a5ed-0e285f8f1614 2024-04-26T22:49:41.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:41.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/69154049-25b2-4b6a-a5ed-0e285f8f1614 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-26T22:49:41.438 INFO:teuthology.orchestra.run.smithi001.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-26T22:49:41.438 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-26T22:49:41.553 INFO:teuthology.orchestra.run.smithi001.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-26T22:49:41.553 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-26T22:49:41.553 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: resolved to: "172.21.15.1:6789,172.21.15.38:6789,172.21.15.186:6789" 2024-04-26T22:49:41.553 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: trying mount with new device syntax: 0@bb8b501a-041d-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/69154049-25b2-4b6a-a5ed-0e285f8f1614 2024-04-26T22:49:41.553 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.1:6789/172.21.15.38:6789/172.21.15.186:6789" will pass to kernel 2024-04-26T22:49:41.554 INFO:teuthology.orchestra.run.smithi001.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T22:49:41.554 INFO:teuthology.orchestra.run.smithi001.stdout: You just mounted a file system that supports labels which does not 2024-04-26T22:49:41.554 INFO:teuthology.orchestra.run.smithi001.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T22:49:41.554 INFO:teuthology.orchestra.run.smithi001.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T22:49:41.554 INFO:teuthology.orchestra.run.smithi001.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T22:49:41.556 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T22:49:41.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T22:49:41.556 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T22:49:41.581 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T22:49:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T22:49:41.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.680 INFO:teuthology.orchestra.run.smithi001.stdout:client24995 2024-04-26T22:49:41.680 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T22:49:41.680 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T22:49:41.697 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T22:49:41.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T22:49:41.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.798 INFO:teuthology.orchestra.run.smithi001.stdout:client24995 2024-04-26T22:49:41.798 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T22:49:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:41 smithi001 ceph-mon[29435]: pgmap v352: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T22:49:41.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:41 smithi038 ceph-mon[32433]: pgmap v352: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:41.856 INFO:teuthology.orchestra.run.smithi001.stdout:client24995 2024-04-26T22:49:41.856 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/bb8b501a-041d-11ef-bc93-c7b262605968.client24995/status') 2024-04-26T22:49:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T22:49:41.915 INFO:teuthology.orchestra.run.smithi001.stdout:instance: client.24995 (3)172.21.15.1:0/385703669 2024-04-26T22:49:41.915 INFO:teuthology.orchestra.run.smithi001.stdout:blocklisted: false 2024-04-26T22:49:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:0+1 records in 2024-04-26T22:49:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:0+1 records out 2024-04-26T22:49:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:69 bytes copied, 2.7553e-05 s, 2.5 MB/s 2024-04-26T22:49:41.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T22:49:41.922 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T22:49:41.932 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T22:49:41.932 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T22:49:41.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-26T22:49:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:41 smithi186 ceph-mon[32774]: pgmap v352: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.035+0000 7fc444e2a640 1 -- 172.21.15.1:0/2503581490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103440 msgr2=0x7fc440105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.035+0000 7fc444e2a640 1 --2- 172.21.15.1:0/2503581490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103440 0x7fc440105830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc430009900 tx=0x7fc43002a410 comp rx=0 tx=0).stop 2024-04-26T22:49:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 -- 172.21.15.1:0/2503581490 shutdown_connections 2024-04-26T22:49:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 --2- 172.21.15.1:0/2503581490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440105d70 0x7fc440108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 --2- 172.21.15.1:0/2503581490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4401086a0 0x7fc44010aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 --2- 172.21.15.1:0/2503581490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc440103440 0x7fc440105830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 -- 172.21.15.1:0/2503581490 >> 172.21.15.1:0/2503581490 conn(0x7fc4400fce50 msgr2=0x7fc4400ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 -- 172.21.15.1:0/2503581490 shutdown_connections 2024-04-26T22:49:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 -- 172.21.15.1:0/2503581490 wait complete. 2024-04-26T22:49:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 Processor -- start 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.036+0000 7fc444e2a640 1 -- start start 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc444e2a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103440 0x7fc440102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc444e2a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440105d70 0x7fc4401013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc444e2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 0x7fc440101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc444e2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010e520 con 0x7fc4401086a0 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc444e2a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010e3a0 con 0x7fc440103440 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc444e2a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010e6a0 con 0x7fc440105d70 2024-04-26T22:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 0x7fc440101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 0x7fc440101930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54636/0 (socket says 172.21.15.1:54636) 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 -- 172.21.15.1:0/3896602321 learned_addr learned my addr 172.21.15.1:0/3896602321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc435d9b640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440105d70 0x7fc4401013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43e59c640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103440 0x7fc440102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 -- 172.21.15.1:0/3896602321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440105d70 msgr2=0x7fc4401013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440105d70 0x7fc4401013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 -- 172.21.15.1:0/3896602321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103440 msgr2=0x7fc440102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103440 0x7fc440102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 -- 172.21.15.1:0/3896602321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4401021f0 con 0x7fc4401086a0 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43e59c640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103440 0x7fc440102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc43ed9d640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 0x7fc440101930 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc43800cad0 tx=0x7fc43800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43800ae20 con 0x7fc4401086a0 2024-04-26T22:49:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.037+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc438005020 con 0x7fc4401086a0 2024-04-26T22:49:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.038+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4401a2dc0 con 0x7fc4401086a0 2024-04-26T22:49:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.038+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43800f6b0 con 0x7fc4401086a0 2024-04-26T22:49:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.038+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4401a3230 con 0x7fc4401086a0 2024-04-26T22:49:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.039+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4400fe940 con 0x7fc4401086a0 2024-04-26T22:49:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.040+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc4380051d0 con 0x7fc4401086a0 2024-04-26T22:49:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.041+0000 7fc437fff640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3c30 0x7fc42c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.041+0000 7fc43e59c640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3c30 0x7fc42c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.041+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc4380d05e0 con 0x7fc4401086a0 2024-04-26T22:49:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.041+0000 7fc43e59c640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3c30 0x7fc42c0a60f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc430006010 tx=0x7fc430000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:42.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.045+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc438014030 con 0x7fc4401086a0 2024-04-26T22:49:42.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.272+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fc44002dad0 con 0x7fc4401086a0 2024-04-26T22:49:42.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.549+0000 7fc437fff640 1 -- 172.21.15.1:0/3896602321 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7fc43809ce90 con 0x7fc4401086a0 2024-04-26T22:49:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3c30 msgr2=0x7fc42c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3c30 0x7fc42c0a60f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc430006010 tx=0x7fc430000bc0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 msgr2=0x7fc440101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 0x7fc440101930 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc43800cad0 tx=0x7fc43800cf90 comp rx=0 tx=0).stop 2024-04-26T22:49:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 shutdown_connections 2024-04-26T22:49:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc42c0a3c30 0x7fc42c0a60f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc440105d70 0x7fc4401013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4401086a0 0x7fc440101930 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 --2- 172.21.15.1:0/3896602321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc440103440 0x7fc440102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 >> 172.21.15.1:0/3896602321 conn(0x7fc4400fce50 msgr2=0x7fc440103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 shutdown_connections 2024-04-26T22:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.552+0000 7fc444e2a640 1 -- 172.21.15.1:0/3896602321 wait complete. 2024-04-26T22:49:42.564 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 -- 172.21.15.1:0/2406479250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 msgr2=0x7f395410adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 --2- 172.21.15.1:0/2406479250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 0x7f395410adf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f394c009ca0 tx=0x7f394c02a8c0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 -- 172.21.15.1:0/2406479250 shutdown_connections 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 --2- 172.21.15.1:0/2406479250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 0x7f395410adf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 --2- 172.21.15.1:0/2406479250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3954103ba0 0x7f3954104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 --2- 172.21.15.1:0/2406479250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39541029b0 0x7f3954102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 -- 172.21.15.1:0/2406479250 >> 172.21.15.1:0/2406479250 conn(0x7f39540fdf20 msgr2=0x7f3954100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 -- 172.21.15.1:0/2406479250 shutdown_connections 2024-04-26T22:49:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.670+0000 7f395b2b1640 1 -- 172.21.15.1:0/2406479250 wait complete. 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 Processor -- start 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 -- start start 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39541029b0 0x7f3954198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3954103ba0 0x7f3954199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 0x7f39541a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f395410e4e0 con 0x7f3954104560 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f395410e360 con 0x7f3954103ba0 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395b2b1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f395410e660 con 0x7f39541029b0 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 0x7f39541a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 0x7f39541a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54656/0 (socket says 172.21.15.1:54656) 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 -- 172.21.15.1:0/3308630788 learned_addr learned my addr 172.21.15.1:0/3308630788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T22:49:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395904d640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39541029b0 0x7f3954198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395884c640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3954103ba0 0x7f3954199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 -- 172.21.15.1:0/3308630788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39541029b0 msgr2=0x7f3954198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39541029b0 0x7f3954198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 -- 172.21.15.1:0/3308630788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3954103ba0 msgr2=0x7f3954199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3954103ba0 0x7f3954199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.671+0000 7f395984e640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39541a0740 con 0x7f3954104560 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.672+0000 7f395984e640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3954104560 0x7f39541a00d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f394c0097b0 tx=0x7f394c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.672+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c004240 con 0x7f3954104560 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.672+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394c0317c0 con 0x7f3954104560 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.672+0000 7f395b2b1640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39541a09d0 con 0x7f3954104560 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.672+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c02fc70 con 0x7f3954104560 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.672+0000 7f395b2b1640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39541a0f20 con 0x7f3954104560 2024-04-26T22:49:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.673+0000 7f395b2b1640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3954109cf0 con 0x7f3954104560 2024-04-26T22:49:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.674+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f394c0043e0 con 0x7f3954104560 2024-04-26T22:49:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.674+0000 7f394a7fc640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 0x7f39400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:49:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.675+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f394c0f1d90 con 0x7f3954104560 2024-04-26T22:49:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.675+0000 7f395904d640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 0x7f39400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:49:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.675+0000 7f395904d640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 0x7f39400a60f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3950005fd0 tx=0x7f395000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:49:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.678+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f394c0be6f0 con 0x7f3954104560 2024-04-26T22:49:42.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:42 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3896602321' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T22:49:42.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:42 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3896602321' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T22:49:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:49:42.892+0000 7f395b2b1640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f3954062d70 con 0x7f39400a3c30 2024-04-26T22:49:42.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:42 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3896602321' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T22:49:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:43 smithi001 ceph-mon[29435]: pgmap v353: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:43 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3896602321' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T22:49:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:43 smithi001 ceph-mon[29435]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T22:49:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:43 smithi038 ceph-mon[32433]: pgmap v353: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:43 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3896602321' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T22:49:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:43 smithi038 ceph-mon[32433]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T22:49:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:43 smithi186 ceph-mon[32774]: pgmap v353: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:49:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:43 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3896602321' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T22:49:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:43 smithi186 ceph-mon[32774]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T22:49:44.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:44 smithi001 ceph-mon[29435]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:44.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:44 smithi038 ceph-mon[32433]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:44.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:44 smithi186 ceph-mon[32774]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T22:49:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:45 smithi001 ceph-mon[29435]: pgmap v354: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:45 smithi038 ceph-mon[32433]: pgmap v354: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:45 smithi186 ceph-mon[32774]: pgmap v354: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:49:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:47 smithi001 ceph-mon[29435]: pgmap v355: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:47 smithi038 ceph-mon[32433]: pgmap v355: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:47 smithi186 ceph-mon[32774]: pgmap v355: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:49:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:49 smithi001 ceph-mon[29435]: pgmap v356: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-26T22:49:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:49 smithi038 ceph-mon[32433]: pgmap v356: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-26T22:49:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:49 smithi186 ceph-mon[32774]: pgmap v356: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-26T22:49:51.640 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:51 smithi001 ceph-mon[29435]: pgmap v357: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:49:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:51 smithi038 ceph-mon[32433]: pgmap v357: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:49:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:51 smithi186 ceph-mon[32774]: pgmap v357: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:49:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:53 smithi001 ceph-mon[29435]: pgmap v358: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-26T22:49:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:53 smithi038 ceph-mon[32433]: pgmap v358: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-26T22:49:53.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:53 smithi186 ceph-mon[32774]: pgmap v358: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-26T22:49:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:55 smithi001 ceph-mon[29435]: pgmap v359: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:49:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:55 smithi038 ceph-mon[32433]: pgmap v359: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:49:55.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:55 smithi186 ceph-mon[32774]: pgmap v359: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-26T22:49:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:57 smithi001 ceph-mon[29435]: pgmap v360: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-26T22:49:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:57 smithi038 ceph-mon[32433]: pgmap v360: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-26T22:49:57.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:57 smithi186 ceph-mon[32774]: pgmap v360: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-26T22:49:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:49:59 smithi001 ceph-mon[29435]: pgmap v361: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-26T22:49:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:49:59 smithi038 ceph-mon[32433]: pgmap v361: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-26T22:49:59.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:49:59 smithi186 ceph-mon[32774]: pgmap v361: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-26T22:50:00.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:00 smithi001 ceph-mon[29435]: overall HEALTH_OK 2024-04-26T22:50:00.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:00 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:50:00.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:00 smithi038 ceph-mon[32433]: overall HEALTH_OK 2024-04-26T22:50:00.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:00 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:50:00.928 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:00 smithi186 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T22:50:00.928 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:00 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:50:01.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:01 smithi038 ceph-mon[32433]: pgmap v362: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:01 smithi001 ceph-mon[29435]: pgmap v362: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:01.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:01 smithi186 ceph-mon[32774]: pgmap v362: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: pgmap v363: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:50:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:50:03.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: pgmap v363: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:50:03.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:50:03.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: pgmap v363: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:50:03.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:50:03.236 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:50:05.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:04 smithi001 ceph-mon[29435]: pgmap v364: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:05.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:04 smithi038 ceph-mon[32433]: pgmap v364: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:05.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:04 smithi186 ceph-mon[32774]: pgmap v364: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:06 smithi001 ceph-mon[29435]: pgmap v365: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:07.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:06 smithi038 ceph-mon[32433]: pgmap v365: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:07.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:06 smithi186 ceph-mon[32774]: pgmap v365: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:09.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:08 smithi001 ceph-mon[29435]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:09.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:08 smithi038 ceph-mon[32433]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:09.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:08 smithi186 ceph-mon[32774]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:10 smithi001 ceph-mon[29435]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:11.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:10 smithi038 ceph-mon[32433]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:11.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:10 smithi186 ceph-mon[32774]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:13.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:12 smithi001 ceph-mon[29435]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:13.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:12 smithi038 ceph-mon[32433]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:13.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:12 smithi186 ceph-mon[32774]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:15.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:14 smithi001 ceph-mon[29435]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:15.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:14 smithi038 ceph-mon[32433]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:15.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:14 smithi186 ceph-mon[32774]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:17.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:16 smithi001 ceph-mon[29435]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:17.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:16 smithi038 ceph-mon[32433]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:17.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:16 smithi186 ceph-mon[32774]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:19.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:18 smithi001 ceph-mon[29435]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:19.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:18 smithi038 ceph-mon[32433]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:19.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:18 smithi186 ceph-mon[32774]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:21.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:20 smithi038 ceph-mon[32433]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:21.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:20 smithi186 ceph-mon[32774]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:21.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:20 smithi001 ceph-mon[29435]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:23.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:22 smithi038 ceph-mon[32433]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:23.148 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:22 smithi001 ceph-mon[29435]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:23.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:22 smithi186 ceph-mon[32774]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:25.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:24 smithi186 ceph-mon[32774]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:25.345 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:24 smithi001 ceph-mon[29435]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:25.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:24 smithi038 ceph-mon[32433]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:27.152 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:26 smithi038 ceph-mon[32433]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:27.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:26 smithi186 ceph-mon[32774]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:27.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:26 smithi001 ceph-mon[29435]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:29.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:28 smithi186 ceph-mon[32774]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:29.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:28 smithi001 ceph-mon[29435]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:29.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:28 smithi038 ceph-mon[32433]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:30 smithi186 ceph-mon[32774]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:30 smithi001 ceph-mon[29435]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:30 smithi038 ceph-mon[32433]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:32 smithi186 ceph-mon[32774]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:32 smithi001 ceph-mon[29435]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:32 smithi038 ceph-mon[32433]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:35.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:34 smithi186 ceph-mon[32774]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:34 smithi001 ceph-mon[29435]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:34 smithi038 ceph-mon[32433]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:37.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:36 smithi186 ceph-mon[32774]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:37.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:36 smithi001 ceph-mon[29435]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:36 smithi038 ceph-mon[32433]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:39.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:38 smithi186 ceph-mon[32774]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:39.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:38 smithi001 ceph-mon[29435]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:38 smithi038 ceph-mon[32433]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:40 smithi186 ceph-mon[32774]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:41.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:40 smithi001 ceph-mon[29435]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:41.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:40 smithi038 ceph-mon[32433]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:42 smithi186 ceph-mon[32774]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:43.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:42 smithi001 ceph-mon[29435]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:43.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:42 smithi038 ceph-mon[32433]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:44 smithi186 ceph-mon[32774]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:45.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:44 smithi001 ceph-mon[29435]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:45.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:44 smithi038 ceph-mon[32433]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:46 smithi186 ceph-mon[32774]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:47.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:46 smithi001 ceph-mon[29435]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:47.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:46 smithi038 ceph-mon[32433]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:48 smithi186 ceph-mon[32774]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:49.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:48 smithi001 ceph-mon[29435]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:49.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:48 smithi038 ceph-mon[32433]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:51.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:50 smithi186 ceph-mon[32774]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:51.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:50 smithi001 ceph-mon[29435]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:50 smithi038 ceph-mon[32433]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:53.148 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:52 smithi001 ceph-mon[29435]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:52 smithi186 ceph-mon[32774]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:52 smithi038 ceph-mon[32433]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:50:55.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:54 smithi186 ceph-mon[32774]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:54 smithi001 ceph-mon[29435]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:54 smithi038 ceph-mon[32433]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:50:57.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:56 smithi186 ceph-mon[32774]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:56 smithi001 ceph-mon[29435]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:57.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:56 smithi038 ceph-mon[32433]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:50:59.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:50:58 smithi186 ceph-mon[32774]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:50:58 smithi001 ceph-mon[29435]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:50:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:50:58 smithi038 ceph-mon[32433]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:01.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:00 smithi186 ceph-mon[32774]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:00 smithi001 ceph-mon[29435]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:00 smithi038 ceph-mon[32433]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:02 smithi186 ceph-mon[32774]: pgmap v393: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:02 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:51:03.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:02 smithi001 ceph-mon[29435]: pgmap v393: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:03.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:02 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:51:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:02 smithi038 ceph-mon[32433]: pgmap v393: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:02 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:51:04.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:51:04.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:04.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:51:04.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:51:04.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:51:05.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:05 smithi186 ceph-mon[32774]: pgmap v394: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:05 smithi001 ceph-mon[29435]: pgmap v394: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:05 smithi038 ceph-mon[32433]: pgmap v394: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:07.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:07 smithi186 ceph-mon[32774]: pgmap v395: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:07 smithi001 ceph-mon[29435]: pgmap v395: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:07 smithi038 ceph-mon[32433]: pgmap v395: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:09.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:09 smithi186 ceph-mon[32774]: pgmap v396: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:09 smithi001 ceph-mon[29435]: pgmap v396: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:09 smithi038 ceph-mon[32433]: pgmap v396: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:11 smithi186 ceph-mon[32774]: pgmap v397: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:11 smithi001 ceph-mon[29435]: pgmap v397: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:11 smithi038 ceph-mon[32433]: pgmap v397: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:13 smithi186 ceph-mon[32774]: pgmap v398: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:13 smithi001 ceph-mon[29435]: pgmap v398: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:13 smithi038 ceph-mon[32433]: pgmap v398: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:15 smithi186 ceph-mon[32774]: pgmap v399: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:15 smithi001 ceph-mon[29435]: pgmap v399: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:15 smithi038 ceph-mon[32433]: pgmap v399: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:17 smithi186 ceph-mon[32774]: pgmap v400: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:17 smithi001 ceph-mon[29435]: pgmap v400: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:17 smithi038 ceph-mon[32433]: pgmap v400: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:19 smithi186 ceph-mon[32774]: pgmap v401: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:19 smithi001 ceph-mon[29435]: pgmap v401: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:19 smithi038 ceph-mon[32433]: pgmap v401: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:21 smithi186 ceph-mon[32774]: pgmap v402: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:21 smithi001 ceph-mon[29435]: pgmap v402: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:21 smithi038 ceph-mon[32433]: pgmap v402: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:23 smithi186 ceph-mon[32774]: pgmap v403: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:23 smithi001 ceph-mon[29435]: pgmap v403: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:23 smithi038 ceph-mon[32433]: pgmap v403: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:25 smithi186 ceph-mon[32774]: pgmap v404: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:25 smithi001 ceph-mon[29435]: pgmap v404: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:25 smithi038 ceph-mon[32433]: pgmap v404: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:27 smithi186 ceph-mon[32774]: pgmap v405: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:27 smithi001 ceph-mon[29435]: pgmap v405: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:27 smithi038 ceph-mon[32433]: pgmap v405: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:29 smithi186 ceph-mon[32774]: pgmap v406: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:29 smithi001 ceph-mon[29435]: pgmap v406: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:29 smithi038 ceph-mon[32433]: pgmap v406: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:31 smithi186 ceph-mon[32774]: pgmap v407: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:31 smithi001 ceph-mon[29435]: pgmap v407: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:31 smithi038 ceph-mon[32433]: pgmap v407: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:33 smithi186 ceph-mon[32774]: pgmap v408: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:33 smithi001 ceph-mon[29435]: pgmap v408: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:33 smithi038 ceph-mon[32433]: pgmap v408: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:35 smithi186 ceph-mon[32774]: pgmap v409: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:35 smithi001 ceph-mon[29435]: pgmap v409: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:35 smithi038 ceph-mon[32433]: pgmap v409: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:37 smithi186 ceph-mon[32774]: pgmap v410: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:37 smithi001 ceph-mon[29435]: pgmap v410: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:37 smithi038 ceph-mon[32433]: pgmap v410: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:39 smithi186 ceph-mon[32774]: pgmap v411: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:39 smithi001 ceph-mon[29435]: pgmap v411: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:39 smithi038 ceph-mon[32433]: pgmap v411: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:41 smithi186 ceph-mon[32774]: pgmap v412: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:41 smithi001 ceph-mon[29435]: pgmap v412: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:41 smithi038 ceph-mon[32433]: pgmap v412: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:43.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:43 smithi186 ceph-mon[32774]: pgmap v413: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:43 smithi001 ceph-mon[29435]: pgmap v413: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:43 smithi038 ceph-mon[32433]: pgmap v413: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:45 smithi186 ceph-mon[32774]: pgmap v414: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:45 smithi001 ceph-mon[29435]: pgmap v414: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:45 smithi038 ceph-mon[32433]: pgmap v414: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:47.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:47 smithi186 ceph-mon[32774]: pgmap v415: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:47 smithi001 ceph-mon[29435]: pgmap v415: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:47 smithi038 ceph-mon[32433]: pgmap v415: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:49 smithi186 ceph-mon[32774]: pgmap v416: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:49 smithi001 ceph-mon[29435]: pgmap v416: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:49 smithi038 ceph-mon[32433]: pgmap v416: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:51.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:51 smithi186 ceph-mon[32774]: pgmap v417: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:51 smithi001 ceph-mon[29435]: pgmap v417: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:51 smithi038 ceph-mon[32433]: pgmap v417: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:53 smithi186 ceph-mon[32774]: pgmap v418: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:53 smithi001 ceph-mon[29435]: pgmap v418: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:53 smithi038 ceph-mon[32433]: pgmap v418: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:51:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:55 smithi186 ceph-mon[32774]: pgmap v419: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:55 smithi001 ceph-mon[29435]: pgmap v419: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:55 smithi038 ceph-mon[32433]: pgmap v419: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:51:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:57 smithi186 ceph-mon[32774]: pgmap v420: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:57 smithi001 ceph-mon[29435]: pgmap v420: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:57 smithi038 ceph-mon[32433]: pgmap v420: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:51:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:51:59 smithi186 ceph-mon[32774]: pgmap v421: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:51:59 smithi001 ceph-mon[29435]: pgmap v421: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:51:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:51:59 smithi038 ceph-mon[32433]: pgmap v421: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:01.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:01 smithi186 ceph-mon[32774]: pgmap v422: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:01 smithi001 ceph-mon[29435]: pgmap v422: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:01 smithi038 ceph-mon[32433]: pgmap v422: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:03 smithi001 ceph-mon[29435]: pgmap v423: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:03 smithi038 ceph-mon[32433]: pgmap v423: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:03 smithi186 ceph-mon[32774]: pgmap v423: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:04.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:04 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:52:04.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:04 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:52:04.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:04 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:52:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:05 smithi001 ceph-mon[29435]: pgmap v424: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:05.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:52:05.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:52:05.850 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:52:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:05 smithi038 ceph-mon[32433]: pgmap v424: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:52:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:52:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:52:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:05 smithi186 ceph-mon[32774]: pgmap v424: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:52:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:52:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:52:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:07 smithi001 ceph-mon[29435]: pgmap v425: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:07 smithi038 ceph-mon[32433]: pgmap v425: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:07 smithi186 ceph-mon[32774]: pgmap v425: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:09 smithi001 ceph-mon[29435]: pgmap v426: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:09 smithi038 ceph-mon[32433]: pgmap v426: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:09 smithi186 ceph-mon[32774]: pgmap v426: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:11 smithi001 ceph-mon[29435]: pgmap v427: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:11 smithi038 ceph-mon[32433]: pgmap v427: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:11 smithi186 ceph-mon[32774]: pgmap v427: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:13 smithi001 ceph-mon[29435]: pgmap v428: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:13 smithi038 ceph-mon[32433]: pgmap v428: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:13 smithi186 ceph-mon[32774]: pgmap v428: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:15 smithi001 ceph-mon[29435]: pgmap v429: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:15 smithi038 ceph-mon[32433]: pgmap v429: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:15.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:15 smithi186 ceph-mon[32774]: pgmap v429: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:17 smithi001 ceph-mon[29435]: pgmap v430: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:17 smithi038 ceph-mon[32433]: pgmap v430: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:17 smithi186 ceph-mon[32774]: pgmap v430: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:19 smithi001 ceph-mon[29435]: pgmap v431: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:19 smithi038 ceph-mon[32433]: pgmap v431: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:19 smithi186 ceph-mon[32774]: pgmap v431: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:21 smithi001 ceph-mon[29435]: pgmap v432: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:21 smithi038 ceph-mon[32433]: pgmap v432: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:21 smithi186 ceph-mon[32774]: pgmap v432: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:23 smithi001 ceph-mon[29435]: pgmap v433: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:23 smithi038 ceph-mon[32433]: pgmap v433: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:23 smithi186 ceph-mon[32774]: pgmap v433: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:25 smithi001 ceph-mon[29435]: pgmap v434: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:25 smithi038 ceph-mon[32433]: pgmap v434: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:25 smithi186 ceph-mon[32774]: pgmap v434: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:27 smithi001 ceph-mon[29435]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:27 smithi038 ceph-mon[32433]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:27 smithi186 ceph-mon[32774]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:29.786 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:29 smithi186 ceph-mon[32774]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:29 smithi001 ceph-mon[29435]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:29 smithi038 ceph-mon[32433]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:31 smithi001 ceph-mon[29435]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:31 smithi038 ceph-mon[32433]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:31 smithi186 ceph-mon[32774]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:33 smithi001 ceph-mon[29435]: pgmap v438: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:33 smithi038 ceph-mon[32433]: pgmap v438: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:33 smithi186 ceph-mon[32774]: pgmap v438: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:35 smithi001 ceph-mon[29435]: pgmap v439: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:35 smithi038 ceph-mon[32433]: pgmap v439: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:35 smithi186 ceph-mon[32774]: pgmap v439: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:37 smithi001 ceph-mon[29435]: pgmap v440: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:37 smithi038 ceph-mon[32433]: pgmap v440: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:37 smithi186 ceph-mon[32774]: pgmap v440: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:39 smithi001 ceph-mon[29435]: pgmap v441: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:39 smithi038 ceph-mon[32433]: pgmap v441: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:39 smithi186 ceph-mon[32774]: pgmap v441: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:41 smithi001 ceph-mon[29435]: pgmap v442: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:41 smithi038 ceph-mon[32433]: pgmap v442: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:41 smithi186 ceph-mon[32774]: pgmap v442: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:52:42.697+0000 7f39497fa640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f393c0034c0 con 0x7f3954104560 2024-04-26T22:52:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:52:42.698+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f393c0034c0 con 0x7f3954104560 2024-04-26T22:52:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:43 smithi001 ceph-mon[29435]: pgmap v443: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:43 smithi038 ceph-mon[32433]: pgmap v443: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:43 smithi186 ceph-mon[32774]: pgmap v443: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:45 smithi001 ceph-mon[29435]: pgmap v444: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:45 smithi038 ceph-mon[32433]: pgmap v444: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:45 smithi186 ceph-mon[32774]: pgmap v444: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:47 smithi001 ceph-mon[29435]: pgmap v445: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:47 smithi038 ceph-mon[32433]: pgmap v445: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:47 smithi186 ceph-mon[32774]: pgmap v445: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:49 smithi001 ceph-mon[29435]: pgmap v446: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:49 smithi038 ceph-mon[32433]: pgmap v446: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:49 smithi186 ceph-mon[32774]: pgmap v446: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:51 smithi001 ceph-mon[29435]: pgmap v447: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:51 smithi038 ceph-mon[32433]: pgmap v447: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:51 smithi186 ceph-mon[32774]: pgmap v447: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:53 smithi001 ceph-mon[29435]: pgmap v448: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:53 smithi038 ceph-mon[32433]: pgmap v448: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:53.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:53 smithi186 ceph-mon[32774]: pgmap v448: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:52:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:55 smithi001 ceph-mon[29435]: pgmap v449: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:55 smithi038 ceph-mon[32433]: pgmap v449: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:55.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:55 smithi186 ceph-mon[32774]: pgmap v449: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:52:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:57 smithi001 ceph-mon[29435]: pgmap v450: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:57 smithi038 ceph-mon[32433]: pgmap v450: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:57.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:57 smithi186 ceph-mon[32774]: pgmap v450: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:52:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:52:59 smithi001 ceph-mon[29435]: pgmap v451: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:52:59 smithi038 ceph-mon[32433]: pgmap v451: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:52:59.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:52:59 smithi186 ceph-mon[32774]: pgmap v451: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:01 smithi001 ceph-mon[29435]: pgmap v452: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:01 smithi038 ceph-mon[32433]: pgmap v452: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:01.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:01 smithi186 ceph-mon[32774]: pgmap v452: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:03 smithi001 ceph-mon[29435]: pgmap v453: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:03 smithi038 ceph-mon[32433]: pgmap v453: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:03 smithi186 ceph-mon[32774]: pgmap v453: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:05 smithi001 ceph-mon[29435]: pgmap v454: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:05 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:53:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:05 smithi038 ceph-mon[32433]: pgmap v454: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:05 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:53:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:05 smithi186 ceph-mon[32774]: pgmap v454: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:05 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:53:06.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:53:06.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:53:06.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:53:06.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:53:06.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:53:06.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:53:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:53:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:53:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:53:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:07 smithi001 ceph-mon[29435]: pgmap v455: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:07 smithi038 ceph-mon[32433]: pgmap v455: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:07 smithi186 ceph-mon[32774]: pgmap v455: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:09 smithi001 ceph-mon[29435]: pgmap v456: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:09 smithi038 ceph-mon[32433]: pgmap v456: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:09 smithi186 ceph-mon[32774]: pgmap v456: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:11 smithi001 ceph-mon[29435]: pgmap v457: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:11 smithi038 ceph-mon[32433]: pgmap v457: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:11 smithi186 ceph-mon[32774]: pgmap v457: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:13 smithi038 ceph-mon[32433]: pgmap v458: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:13 smithi186 ceph-mon[32774]: pgmap v458: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:14.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:13 smithi001 ceph-mon[29435]: pgmap v458: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:15 smithi038 ceph-mon[32433]: pgmap v459: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:15.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:15 smithi186 ceph-mon[32774]: pgmap v459: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:16.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:15 smithi001 ceph-mon[29435]: pgmap v459: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:17 smithi186 ceph-mon[32774]: pgmap v460: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:18.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:17 smithi001 ceph-mon[29435]: pgmap v460: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:18.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:17 smithi038 ceph-mon[32433]: pgmap v460: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:19 smithi186 ceph-mon[32774]: pgmap v461: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:20.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:19 smithi001 ceph-mon[29435]: pgmap v461: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:20.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:19 smithi038 ceph-mon[32433]: pgmap v461: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:21 smithi186 ceph-mon[32774]: pgmap v462: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:22.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:21 smithi001 ceph-mon[29435]: pgmap v462: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:22.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:21 smithi038 ceph-mon[32433]: pgmap v462: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:23.898 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:23 smithi001 ceph-mon[29435]: pgmap v463: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:23 smithi186 ceph-mon[32774]: pgmap v463: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:24.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:23 smithi038 ceph-mon[32433]: pgmap v463: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:25 smithi186 ceph-mon[32774]: pgmap v464: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:26.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:25 smithi001 ceph-mon[29435]: pgmap v464: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:26.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:25 smithi038 ceph-mon[32433]: pgmap v464: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:27.903 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:27 smithi038 ceph-mon[32433]: pgmap v465: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:27 smithi186 ceph-mon[32774]: pgmap v465: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:28.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:27 smithi001 ceph-mon[29435]: pgmap v465: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:29.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:29 smithi186 ceph-mon[32774]: pgmap v466: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:29 smithi001 ceph-mon[29435]: pgmap v466: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:30.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:29 smithi038 ceph-mon[32433]: pgmap v466: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:31 smithi186 ceph-mon[32774]: pgmap v467: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:32.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:31 smithi001 ceph-mon[29435]: pgmap v467: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:32.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:31 smithi038 ceph-mon[32433]: pgmap v467: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:33 smithi186 ceph-mon[32774]: pgmap v468: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:34.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:33 smithi001 ceph-mon[29435]: pgmap v468: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:34.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:33 smithi038 ceph-mon[32433]: pgmap v468: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:35 smithi186 ceph-mon[32774]: pgmap v469: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:36.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:35 smithi001 ceph-mon[29435]: pgmap v469: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:35 smithi038 ceph-mon[32433]: pgmap v469: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:37 smithi186 ceph-mon[32774]: pgmap v470: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:38.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:37 smithi001 ceph-mon[29435]: pgmap v470: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:38.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:37 smithi038 ceph-mon[32433]: pgmap v470: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:39 smithi186 ceph-mon[32774]: pgmap v471: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:39 smithi001 ceph-mon[29435]: pgmap v471: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:39 smithi038 ceph-mon[32433]: pgmap v471: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:53:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:41 smithi186 ceph-mon[32774]: pgmap v472: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:41 smithi001 ceph-mon[29435]: pgmap v472: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:41 smithi038 ceph-mon[32433]: pgmap v472: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:43 smithi186 ceph-mon[32774]: pgmap v473: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:43 smithi001 ceph-mon[29435]: pgmap v473: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:43 smithi038 ceph-mon[32433]: pgmap v473: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:53:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:45 smithi186 ceph-mon[32774]: pgmap v474: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:45 smithi001 ceph-mon[29435]: pgmap v474: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:45 smithi038 ceph-mon[32433]: pgmap v474: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:53:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:47 smithi186 ceph-mon[32774]: pgmap v475: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:47 smithi001 ceph-mon[29435]: pgmap v475: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:47 smithi038 ceph-mon[32433]: pgmap v475: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:53:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:49 smithi186 ceph-mon[32774]: pgmap v476: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-26T22:53:50.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:49 smithi001 ceph-mon[29435]: pgmap v476: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-26T22:53:50.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:49 smithi038 ceph-mon[32433]: pgmap v476: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-26T22:53:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:51 smithi186 ceph-mon[32774]: pgmap v477: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-26T22:53:52.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:51 smithi001 ceph-mon[29435]: pgmap v477: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-26T22:53:52.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:51 smithi038 ceph-mon[32433]: pgmap v477: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-26T22:53:53.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:53 smithi186 ceph-mon[32774]: pgmap v478: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-26T22:53:54.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:53 smithi001 ceph-mon[29435]: pgmap v478: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-26T22:53:54.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:53 smithi038 ceph-mon[32433]: pgmap v478: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-26T22:53:55.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:55 smithi186 ceph-mon[32774]: pgmap v479: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-26T22:53:56.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:55 smithi001 ceph-mon[29435]: pgmap v479: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-26T22:53:56.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:55 smithi038 ceph-mon[32433]: pgmap v479: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-26T22:53:57.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:57 smithi186 ceph-mon[32774]: pgmap v480: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-26T22:53:58.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:57 smithi001 ceph-mon[29435]: pgmap v480: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-26T22:53:58.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:57 smithi038 ceph-mon[32433]: pgmap v480: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-26T22:53:59.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:53:59 smithi186 ceph-mon[32774]: pgmap v481: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-26T22:54:00.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:53:59 smithi001 ceph-mon[29435]: pgmap v481: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-26T22:54:00.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:53:59 smithi038 ceph-mon[32433]: pgmap v481: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-26T22:54:01.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:01 smithi186 ceph-mon[32774]: pgmap v482: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:02.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:01 smithi001 ceph-mon[29435]: pgmap v482: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:02.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:01 smithi038 ceph-mon[32433]: pgmap v482: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:03 smithi186 ceph-mon[32774]: pgmap v483: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:03 smithi001 ceph-mon[29435]: pgmap v483: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:03 smithi038 ceph-mon[32433]: pgmap v483: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:05 smithi186 ceph-mon[32774]: pgmap v484: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:06.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:05 smithi001 ceph-mon[29435]: pgmap v484: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:06.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:05 smithi038 ceph-mon[32433]: pgmap v484: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:54:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:54:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:54:06.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:54:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:54:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:54:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:54:07.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:54:07.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:54:07.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:54:07.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:54:07.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:06 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:54:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:07 smithi186 ceph-mon[32774]: pgmap v485: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:07 smithi001 ceph-mon[29435]: pgmap v485: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:08.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:07 smithi038 ceph-mon[32433]: pgmap v485: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:09 smithi186 ceph-mon[32774]: pgmap v486: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:09 smithi001 ceph-mon[29435]: pgmap v486: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:09 smithi038 ceph-mon[32433]: pgmap v486: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:11 smithi186 ceph-mon[32774]: pgmap v487: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:12.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:11 smithi001 ceph-mon[29435]: pgmap v487: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:12.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:11 smithi038 ceph-mon[32433]: pgmap v487: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:13 smithi186 ceph-mon[32774]: pgmap v488: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:14.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:13 smithi001 ceph-mon[29435]: pgmap v488: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:14.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:13 smithi038 ceph-mon[32433]: pgmap v488: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:15.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:14 smithi001 ceph-mon[29435]: pgmap v489: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:15.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:14 smithi038 ceph-mon[32433]: pgmap v489: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:15.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:14 smithi186 ceph-mon[32774]: pgmap v489: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:17.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:16 smithi186 ceph-mon[32774]: pgmap v490: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:17.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:16 smithi001 ceph-mon[29435]: pgmap v490: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:17.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:16 smithi038 ceph-mon[32433]: pgmap v490: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:19.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:18 smithi186 ceph-mon[32774]: pgmap v491: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:19.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:18 smithi001 ceph-mon[29435]: pgmap v491: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:19.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:18 smithi038 ceph-mon[32433]: pgmap v491: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:21.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:20 smithi186 ceph-mon[32774]: pgmap v492: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:21.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:20 smithi001 ceph-mon[29435]: pgmap v492: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:21.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:20 smithi038 ceph-mon[32433]: pgmap v492: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:23.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:22 smithi186 ceph-mon[32774]: pgmap v493: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:23.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:22 smithi001 ceph-mon[29435]: pgmap v493: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:23.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:22 smithi038 ceph-mon[32433]: pgmap v493: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:25.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:24 smithi186 ceph-mon[32774]: pgmap v494: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:25.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:24 smithi001 ceph-mon[29435]: pgmap v494: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:25.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:24 smithi038 ceph-mon[32433]: pgmap v494: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:27.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:26 smithi186 ceph-mon[32774]: pgmap v495: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:27.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:26 smithi001 ceph-mon[29435]: pgmap v495: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:27.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:26 smithi038 ceph-mon[32433]: pgmap v495: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:29.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:28 smithi186 ceph-mon[32774]: pgmap v496: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:29.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:28 smithi001 ceph-mon[29435]: pgmap v496: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:29.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:28 smithi038 ceph-mon[32433]: pgmap v496: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:31.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:30 smithi186 ceph-mon[32774]: pgmap v497: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:30 smithi001 ceph-mon[29435]: pgmap v497: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:30 smithi038 ceph-mon[32433]: pgmap v497: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:33.238 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:32 smithi186 ceph-mon[32774]: pgmap v498: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:32 smithi001 ceph-mon[29435]: pgmap v498: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:32 smithi038 ceph-mon[32433]: pgmap v498: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:35.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:34 smithi186 ceph-mon[32774]: pgmap v499: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:34 smithi001 ceph-mon[29435]: pgmap v499: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:34 smithi038 ceph-mon[32433]: pgmap v499: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:37.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:36 smithi186 ceph-mon[32774]: pgmap v500: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:37.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:36 smithi001 ceph-mon[29435]: pgmap v500: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:36 smithi038 ceph-mon[32433]: pgmap v500: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:39.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:38 smithi186 ceph-mon[32774]: pgmap v501: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:39.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:38 smithi001 ceph-mon[29435]: pgmap v501: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:38 smithi038 ceph-mon[32433]: pgmap v501: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:41.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:40 smithi186 ceph-mon[32774]: pgmap v502: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:41.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:40 smithi001 ceph-mon[29435]: pgmap v502: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:41.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:40 smithi038 ceph-mon[32433]: pgmap v502: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:43.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:42 smithi186 ceph-mon[32774]: pgmap v503: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:43.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:42 smithi001 ceph-mon[29435]: pgmap v503: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:43.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:42 smithi038 ceph-mon[32433]: pgmap v503: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:45.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:44 smithi186 ceph-mon[32774]: pgmap v504: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:45.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:44 smithi001 ceph-mon[29435]: pgmap v504: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:45.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:44 smithi038 ceph-mon[32433]: pgmap v504: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:46 smithi186 ceph-mon[32774]: pgmap v505: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:47.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:46 smithi001 ceph-mon[29435]: pgmap v505: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:47.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:46 smithi038 ceph-mon[32433]: pgmap v505: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:48 smithi186 ceph-mon[32774]: pgmap v506: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:49.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:48 smithi001 ceph-mon[29435]: pgmap v506: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:49.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:48 smithi038 ceph-mon[32433]: pgmap v506: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:51.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:50 smithi186 ceph-mon[32774]: pgmap v507: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:51.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:50 smithi001 ceph-mon[29435]: pgmap v507: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:50 smithi038 ceph-mon[32433]: pgmap v507: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:53.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:52 smithi186 ceph-mon[32774]: pgmap v508: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:53.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:52 smithi001 ceph-mon[29435]: pgmap v508: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:52 smithi038 ceph-mon[32433]: pgmap v508: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:54:55.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:54 smithi186 ceph-mon[32774]: pgmap v509: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:54 smithi001 ceph-mon[29435]: pgmap v509: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:54 smithi038 ceph-mon[32433]: pgmap v509: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:54:57.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:56 smithi186 ceph-mon[32774]: pgmap v510: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:56 smithi001 ceph-mon[29435]: pgmap v510: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:57.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:56 smithi038 ceph-mon[32433]: pgmap v510: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:54:59.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:54:58 smithi186 ceph-mon[32774]: pgmap v511: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:54:58 smithi001 ceph-mon[29435]: pgmap v511: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:54:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:54:58 smithi038 ceph-mon[32433]: pgmap v511: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:00 smithi001 ceph-mon[29435]: pgmap v512: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:00 smithi038 ceph-mon[32433]: pgmap v512: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:00 smithi186 ceph-mon[32774]: pgmap v512: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:03.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:02 smithi001 ceph-mon[29435]: pgmap v513: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:02 smithi038 ceph-mon[32433]: pgmap v513: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:03.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:02 smithi186 ceph-mon[32774]: pgmap v513: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:05.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:04 smithi001 ceph-mon[29435]: pgmap v514: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:05.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:04 smithi038 ceph-mon[32433]: pgmap v514: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:04 smithi186 ceph-mon[32774]: pgmap v514: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:07.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:06 smithi001 ceph-mon[29435]: pgmap v515: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T22:55:07.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:06 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:55:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:07 smithi038 ceph-mon[32433]: pgmap v515: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T22:55:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:07 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:55:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:06 smithi186 ceph-mon[32774]: pgmap v515: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T22:55:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:06 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:55:08.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:55:08.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:55:08.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:07 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:55:08.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:55:08.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:55:08.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:55:08.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:55:08.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:55:08.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:07 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:55:09.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:09 smithi186 ceph-mon[32774]: pgmap v516: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:09 smithi001 ceph-mon[29435]: pgmap v516: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:09 smithi038 ceph-mon[32433]: pgmap v516: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:11.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:11 smithi186 ceph-mon[32774]: pgmap v517: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:11 smithi001 ceph-mon[29435]: pgmap v517: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:11 smithi038 ceph-mon[32433]: pgmap v517: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:13 smithi186 ceph-mon[32774]: pgmap v518: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:13 smithi001 ceph-mon[29435]: pgmap v518: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:13 smithi038 ceph-mon[32433]: pgmap v518: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:15 smithi186 ceph-mon[32774]: pgmap v519: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:15 smithi001 ceph-mon[29435]: pgmap v519: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:15 smithi038 ceph-mon[32433]: pgmap v519: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:17 smithi186 ceph-mon[32774]: pgmap v520: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:17 smithi001 ceph-mon[29435]: pgmap v520: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:17 smithi038 ceph-mon[32433]: pgmap v520: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:19 smithi186 ceph-mon[32774]: pgmap v521: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:19 smithi001 ceph-mon[29435]: pgmap v521: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:19 smithi038 ceph-mon[32433]: pgmap v521: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:21 smithi186 ceph-mon[32774]: pgmap v522: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:21 smithi001 ceph-mon[29435]: pgmap v522: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:21 smithi038 ceph-mon[32433]: pgmap v522: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:23 smithi186 ceph-mon[32774]: pgmap v523: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:23 smithi001 ceph-mon[29435]: pgmap v523: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:23 smithi038 ceph-mon[32433]: pgmap v523: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:25 smithi186 ceph-mon[32774]: pgmap v524: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:25 smithi001 ceph-mon[29435]: pgmap v524: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:25 smithi038 ceph-mon[32433]: pgmap v524: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:27 smithi001 ceph-mon[29435]: pgmap v525: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:27 smithi038 ceph-mon[32433]: pgmap v525: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:27 smithi186 ceph-mon[32774]: pgmap v525: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:29.787 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:29 smithi186 ceph-mon[32774]: pgmap v526: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:29 smithi001 ceph-mon[29435]: pgmap v526: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:29 smithi038 ceph-mon[32433]: pgmap v526: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:31 smithi001 ceph-mon[29435]: pgmap v527: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:31 smithi038 ceph-mon[32433]: pgmap v527: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:31 smithi186 ceph-mon[32774]: pgmap v527: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:33 smithi001 ceph-mon[29435]: pgmap v528: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:33 smithi038 ceph-mon[32433]: pgmap v528: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:33 smithi186 ceph-mon[32774]: pgmap v528: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:35 smithi001 ceph-mon[29435]: pgmap v529: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:35 smithi038 ceph-mon[32433]: pgmap v529: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:35 smithi186 ceph-mon[32774]: pgmap v529: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:37 smithi001 ceph-mon[29435]: pgmap v530: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:37 smithi038 ceph-mon[32433]: pgmap v530: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:37 smithi186 ceph-mon[32774]: pgmap v530: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:39 smithi001 ceph-mon[29435]: pgmap v531: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:39 smithi038 ceph-mon[32433]: pgmap v531: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:39 smithi186 ceph-mon[32774]: pgmap v531: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:41 smithi001 ceph-mon[29435]: pgmap v532: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:41 smithi038 ceph-mon[32433]: pgmap v532: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:41 smithi186 ceph-mon[32774]: pgmap v532: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:55:42.723+0000 7f39497fa640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f393c005850 con 0x7f3954104560 2024-04-26T22:55:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:55:42.723+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f393c005850 con 0x7f3954104560 2024-04-26T22:55:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:43 smithi001 ceph-mon[29435]: pgmap v533: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:43 smithi038 ceph-mon[32433]: pgmap v533: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:43 smithi186 ceph-mon[32774]: pgmap v533: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:45 smithi001 ceph-mon[29435]: pgmap v534: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:45 smithi038 ceph-mon[32433]: pgmap v534: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:45 smithi186 ceph-mon[32774]: pgmap v534: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T22:55:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:47 smithi001 ceph-mon[29435]: pgmap v535: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T22:55:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:47 smithi038 ceph-mon[32433]: pgmap v535: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T22:55:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:47 smithi186 ceph-mon[32774]: pgmap v535: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T22:55:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:49 smithi001 ceph-mon[29435]: pgmap v536: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:49 smithi038 ceph-mon[32433]: pgmap v536: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:49 smithi186 ceph-mon[32774]: pgmap v536: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T22:55:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:51 smithi001 ceph-mon[29435]: pgmap v537: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:51 smithi038 ceph-mon[32433]: pgmap v537: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:51 smithi186 ceph-mon[32774]: pgmap v537: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:53 smithi001 ceph-mon[29435]: pgmap v538: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:53 smithi038 ceph-mon[32433]: pgmap v538: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:53.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:53 smithi186 ceph-mon[32774]: pgmap v538: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:55:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:55 smithi001 ceph-mon[29435]: pgmap v539: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:55 smithi038 ceph-mon[32433]: pgmap v539: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:55.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:55 smithi186 ceph-mon[32774]: pgmap v539: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:55:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:57 smithi001 ceph-mon[29435]: pgmap v540: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:57 smithi038 ceph-mon[32433]: pgmap v540: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:57.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:57 smithi186 ceph-mon[32774]: pgmap v540: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:55:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:55:59 smithi001 ceph-mon[29435]: pgmap v541: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:55:59 smithi038 ceph-mon[32433]: pgmap v541: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:55:59.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:55:59 smithi186 ceph-mon[32774]: pgmap v541: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:01 smithi001 ceph-mon[29435]: pgmap v542: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:01 smithi038 ceph-mon[32433]: pgmap v542: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:01.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:01 smithi186 ceph-mon[32774]: pgmap v542: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:03 smithi001 ceph-mon[29435]: pgmap v543: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:03 smithi038 ceph-mon[32433]: pgmap v543: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:03 smithi186 ceph-mon[32774]: pgmap v543: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:05 smithi001 ceph-mon[29435]: pgmap v544: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:05 smithi038 ceph-mon[32433]: pgmap v544: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:05 smithi186 ceph-mon[32774]: pgmap v544: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:07 smithi001 ceph-mon[29435]: pgmap v545: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:07 smithi038 ceph-mon[32433]: pgmap v545: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:07 smithi186 ceph-mon[32774]: pgmap v545: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:56:08.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:56:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:56:08.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:56:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:56:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:56:08.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:56:08.852 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:56:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:56:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:09 smithi001 ceph-mon[29435]: pgmap v546: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:09 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi001 to 177.2M 2024-04-26T22:56:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:09 smithi001 ceph-mon[29435]: Unable to set osd_memory_target on smithi001 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T22:56:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:09 smithi038 ceph-mon[32433]: pgmap v546: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:09 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi001 to 177.2M 2024-04-26T22:56:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:09 smithi038 ceph-mon[32433]: Unable to set osd_memory_target on smithi001 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T22:56:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:09 smithi186 ceph-mon[32774]: pgmap v546: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:09 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi001 to 177.2M 2024-04-26T22:56:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:09 smithi186 ceph-mon[32774]: Unable to set osd_memory_target on smithi001 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T22:56:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:11 smithi001 ceph-mon[29435]: pgmap v547: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:11 smithi038 ceph-mon[32433]: pgmap v547: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:11 smithi186 ceph-mon[32774]: pgmap v547: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:13 smithi001 ceph-mon[29435]: pgmap v548: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:13 smithi038 ceph-mon[32433]: pgmap v548: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:13 smithi186 ceph-mon[32774]: pgmap v548: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:15 smithi001 ceph-mon[29435]: pgmap v549: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:15 smithi038 ceph-mon[32433]: pgmap v549: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:15.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:15 smithi186 ceph-mon[32774]: pgmap v549: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:17 smithi001 ceph-mon[29435]: pgmap v550: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:17 smithi038 ceph-mon[32433]: pgmap v550: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:17 smithi186 ceph-mon[32774]: pgmap v550: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:19 smithi001 ceph-mon[29435]: pgmap v551: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:19 smithi038 ceph-mon[32433]: pgmap v551: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:19 smithi186 ceph-mon[32774]: pgmap v551: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:21 smithi001 ceph-mon[29435]: pgmap v552: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:21 smithi038 ceph-mon[32433]: pgmap v552: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:21 smithi186 ceph-mon[32774]: pgmap v552: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:23 smithi001 ceph-mon[29435]: pgmap v553: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:23 smithi038 ceph-mon[32433]: pgmap v553: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:23 smithi186 ceph-mon[32774]: pgmap v553: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:25 smithi001 ceph-mon[29435]: pgmap v554: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:25 smithi038 ceph-mon[32433]: pgmap v554: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:25 smithi186 ceph-mon[32774]: pgmap v554: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:27 smithi001 ceph-mon[29435]: pgmap v555: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:27 smithi038 ceph-mon[32433]: pgmap v555: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:27 smithi186 ceph-mon[32774]: pgmap v555: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:29 smithi038 ceph-mon[32433]: pgmap v556: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:29.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:29 smithi186 ceph-mon[32774]: pgmap v556: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:29 smithi001 ceph-mon[29435]: pgmap v556: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:31 smithi038 ceph-mon[32433]: pgmap v557: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:31 smithi186 ceph-mon[32774]: pgmap v557: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:32.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:31 smithi001 ceph-mon[29435]: pgmap v557: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:33 smithi186 ceph-mon[32774]: pgmap v558: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:34.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:33 smithi001 ceph-mon[29435]: pgmap v558: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:34.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:33 smithi038 ceph-mon[32433]: pgmap v558: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:35 smithi186 ceph-mon[32774]: pgmap v559: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:36.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:35 smithi001 ceph-mon[29435]: pgmap v559: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:36.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:35 smithi038 ceph-mon[32433]: pgmap v559: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:37 smithi186 ceph-mon[32774]: pgmap v560: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:38.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:37 smithi001 ceph-mon[29435]: pgmap v560: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:38.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:37 smithi038 ceph-mon[32433]: pgmap v560: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:39 smithi186 ceph-mon[32774]: pgmap v561: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:39 smithi001 ceph-mon[29435]: pgmap v561: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:39 smithi038 ceph-mon[32433]: pgmap v561: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:41 smithi186 ceph-mon[32774]: pgmap v562: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:41 smithi001 ceph-mon[29435]: pgmap v562: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:41 smithi038 ceph-mon[32433]: pgmap v562: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:43 smithi186 ceph-mon[32774]: pgmap v563: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:43 smithi001 ceph-mon[29435]: pgmap v563: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:43 smithi038 ceph-mon[32433]: pgmap v563: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:45 smithi186 ceph-mon[32774]: pgmap v564: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:45 smithi001 ceph-mon[29435]: pgmap v564: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:45 smithi038 ceph-mon[32433]: pgmap v564: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:47 smithi186 ceph-mon[32774]: pgmap v565: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:48.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:47 smithi001 ceph-mon[29435]: pgmap v565: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:48.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:47 smithi038 ceph-mon[32433]: pgmap v565: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:49 smithi186 ceph-mon[32774]: pgmap v566: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:50.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:49 smithi001 ceph-mon[29435]: pgmap v566: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:50.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:49 smithi038 ceph-mon[32433]: pgmap v566: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:56:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:51 smithi186 ceph-mon[32774]: pgmap v567: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:52.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:51 smithi001 ceph-mon[29435]: pgmap v567: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:52.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:51 smithi038 ceph-mon[32433]: pgmap v567: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:53.898 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:53 smithi001 ceph-mon[29435]: pgmap v568: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:53.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:53 smithi186 ceph-mon[32774]: pgmap v568: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:54.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:53 smithi038 ceph-mon[32433]: pgmap v568: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:56:55.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:55 smithi186 ceph-mon[32774]: pgmap v569: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:56.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:55 smithi001 ceph-mon[29435]: pgmap v569: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:56.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:55 smithi038 ceph-mon[32433]: pgmap v569: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:56:57.904 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:57 smithi038 ceph-mon[32433]: pgmap v570: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:57.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:57 smithi186 ceph-mon[32774]: pgmap v570: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:58.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:57 smithi001 ceph-mon[29435]: pgmap v570: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:56:59.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:56:59 smithi186 ceph-mon[32774]: pgmap v571: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:00.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:56:59 smithi001 ceph-mon[29435]: pgmap v571: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:00.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:56:59 smithi038 ceph-mon[32433]: pgmap v571: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:01.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:01 smithi186 ceph-mon[32774]: pgmap v572: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:02.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:01 smithi001 ceph-mon[29435]: pgmap v572: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:02.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:01 smithi038 ceph-mon[32433]: pgmap v572: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:03 smithi186 ceph-mon[32774]: pgmap v573: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:04.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:03 smithi001 ceph-mon[29435]: pgmap v573: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:04.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:03 smithi038 ceph-mon[32433]: pgmap v573: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:05 smithi186 ceph-mon[32774]: pgmap v574: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:06.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:05 smithi001 ceph-mon[29435]: pgmap v574: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:06.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:05 smithi038 ceph-mon[32433]: pgmap v574: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:07 smithi186 ceph-mon[32774]: pgmap v575: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:08.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:07 smithi001 ceph-mon[29435]: pgmap v575: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:08.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:07 smithi038 ceph-mon[32433]: pgmap v575: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:08.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:08 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:57:09.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:08 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:57:09.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:08 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: pgmap v576: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:09.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:57:09.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:57:09.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:09 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:57:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: pgmap v576: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:57:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:57:10.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:09 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: pgmap v576: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:10.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:57:10.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:57:10.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:09 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:57:10.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:10 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T22:57:10.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:10 smithi186 ceph-mon[32774]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T22:57:10.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:10 smithi186 ceph-mon[32774]: Unable to set osd_memory_target on smithi186 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T22:57:10.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:10 smithi186 ceph-mon[32774]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T22:57:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:10 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T22:57:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:10 smithi001 ceph-mon[29435]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T22:57:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:10 smithi001 ceph-mon[29435]: Unable to set osd_memory_target on smithi186 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T22:57:11.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:10 smithi001 ceph-mon[29435]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T22:57:11.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:10 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T22:57:11.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:10 smithi038 ceph-mon[32433]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T22:57:11.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:10 smithi038 ceph-mon[32433]: Unable to set osd_memory_target on smithi186 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T22:57:11.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:10 smithi038 ceph-mon[32433]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T22:57:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:11 smithi186 ceph-mon[32774]: pgmap v577: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:12.097 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:11 smithi001 ceph-mon[29435]: pgmap v577: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:12.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:11 smithi038 ceph-mon[32433]: pgmap v577: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:13 smithi186 ceph-mon[32774]: pgmap v578: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:14.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:13 smithi001 ceph-mon[29435]: pgmap v578: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:14.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:13 smithi038 ceph-mon[32433]: pgmap v578: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:15.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:15 smithi186 ceph-mon[32774]: pgmap v579: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:16.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:15 smithi001 ceph-mon[29435]: pgmap v579: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:16.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:15 smithi038 ceph-mon[32433]: pgmap v579: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:17 smithi186 ceph-mon[32774]: pgmap v580: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:18.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:17 smithi001 ceph-mon[29435]: pgmap v580: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:18.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:17 smithi038 ceph-mon[32433]: pgmap v580: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:19 smithi186 ceph-mon[32774]: pgmap v581: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:20.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:19 smithi001 ceph-mon[29435]: pgmap v581: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:20.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:19 smithi038 ceph-mon[32433]: pgmap v581: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:21.986 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:21 smithi186 ceph-mon[32774]: pgmap v582: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:22.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:21 smithi001 ceph-mon[29435]: pgmap v582: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:22.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:21 smithi038 ceph-mon[32433]: pgmap v582: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:23 smithi186 ceph-mon[32774]: pgmap v583: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:24.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:23 smithi001 ceph-mon[29435]: pgmap v583: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:24.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:23 smithi038 ceph-mon[32433]: pgmap v583: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:25 smithi186 ceph-mon[32774]: pgmap v584: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:26.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:25 smithi001 ceph-mon[29435]: pgmap v584: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:26.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:25 smithi038 ceph-mon[32433]: pgmap v584: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:27 smithi186 ceph-mon[32774]: pgmap v585: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:28.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:27 smithi001 ceph-mon[29435]: pgmap v585: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:28.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:27 smithi038 ceph-mon[32433]: pgmap v585: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:29.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:29 smithi186 ceph-mon[32774]: pgmap v586: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:29 smithi001 ceph-mon[29435]: pgmap v586: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:30.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:29 smithi038 ceph-mon[32433]: pgmap v586: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:31 smithi186 ceph-mon[32774]: pgmap v587: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:32.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:31 smithi001 ceph-mon[29435]: pgmap v587: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:32.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:31 smithi038 ceph-mon[32433]: pgmap v587: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:33.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:33 smithi186 ceph-mon[32774]: pgmap v588: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:34.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:33 smithi001 ceph-mon[29435]: pgmap v588: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:34.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:33 smithi038 ceph-mon[32433]: pgmap v588: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:35.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:35 smithi186 ceph-mon[32774]: pgmap v589: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:36.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:35 smithi001 ceph-mon[29435]: pgmap v589: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:36.100 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:35 smithi038 ceph-mon[32433]: pgmap v589: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:37 smithi186 ceph-mon[32774]: pgmap v590: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:38.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:37 smithi001 ceph-mon[29435]: pgmap v590: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:38.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:37 smithi038 ceph-mon[32433]: pgmap v590: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:39 smithi186 ceph-mon[32774]: pgmap v591: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:40.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:39 smithi001 ceph-mon[29435]: pgmap v591: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:40.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:39 smithi038 ceph-mon[32433]: pgmap v591: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:41 smithi186 ceph-mon[32774]: pgmap v592: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:42.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:41 smithi001 ceph-mon[29435]: pgmap v592: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:42.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:41 smithi038 ceph-mon[32433]: pgmap v592: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:43 smithi186 ceph-mon[32774]: pgmap v593: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:44.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:43 smithi001 ceph-mon[29435]: pgmap v593: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:44.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:43 smithi038 ceph-mon[32433]: pgmap v593: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:45 smithi186 ceph-mon[32774]: pgmap v594: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:46.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:45 smithi001 ceph-mon[29435]: pgmap v594: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:46.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:45 smithi038 ceph-mon[32433]: pgmap v594: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:47.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:46 smithi001 ceph-mon[29435]: pgmap v595: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:47.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:46 smithi038 ceph-mon[32433]: pgmap v595: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:47.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:46 smithi186 ceph-mon[32774]: pgmap v595: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:49.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:48 smithi186 ceph-mon[32774]: pgmap v596: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:49.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:48 smithi001 ceph-mon[29435]: pgmap v596: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:49.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:48 smithi038 ceph-mon[32433]: pgmap v596: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:51.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:50 smithi001 ceph-mon[29435]: pgmap v597: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:50 smithi038 ceph-mon[32433]: pgmap v597: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:51.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:50 smithi186 ceph-mon[32774]: pgmap v597: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:53.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:52 smithi001 ceph-mon[29435]: pgmap v598: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:52 smithi038 ceph-mon[32433]: pgmap v598: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:53.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:52 smithi186 ceph-mon[32774]: pgmap v598: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:57:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:54 smithi001 ceph-mon[29435]: pgmap v599: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:54 smithi038 ceph-mon[32433]: pgmap v599: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:54 smithi186 ceph-mon[32774]: pgmap v599: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:57:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:56 smithi001 ceph-mon[29435]: pgmap v600: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:57.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:57 smithi038 ceph-mon[32433]: pgmap v600: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:57.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:56 smithi186 ceph-mon[32774]: pgmap v600: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:57:57.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:57:57.741+0000 7f39497fa640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f393c005b70 con 0x7f3954104560 2024-04-26T22:57:57.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:57:57.742+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f393c005b70 con 0x7f3954104560 2024-04-26T22:57:59.287 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:57:58 smithi186 ceph-mon[32774]: pgmap v601: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:57:58 smithi001 ceph-mon[29435]: pgmap v601: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:57:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:57:59 smithi038 ceph-mon[32433]: pgmap v601: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:01 smithi001 ceph-mon[29435]: pgmap v602: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:01 smithi038 ceph-mon[32433]: pgmap v602: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:01 smithi186 ceph-mon[32774]: pgmap v602: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:03.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:03 smithi001 ceph-mon[29435]: pgmap v603: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:03 smithi038 ceph-mon[32433]: pgmap v603: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:03.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:03 smithi186 ceph-mon[32774]: pgmap v603: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:05.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:05 smithi001 ceph-mon[29435]: pgmap v604: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:05.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:05 smithi038 ceph-mon[32433]: pgmap v604: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:05 smithi186 ceph-mon[32774]: pgmap v604: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:07.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:07 smithi001 ceph-mon[29435]: pgmap v605: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:07 smithi038 ceph-mon[32433]: pgmap v605: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:07 smithi186 ceph-mon[32774]: pgmap v605: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:09.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:09 smithi001 ceph-mon[29435]: pgmap v606: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:09.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:09 smithi038 ceph-mon[32433]: pgmap v606: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:09 smithi186 ceph-mon[32774]: pgmap v606: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:10.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:58:10.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:58:10.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:58:10.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:58:10.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:58:10.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:58:10.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:58:10.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:58:10.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:58:10.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:58:10.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:58:10.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:58:11.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:11 smithi001 ceph-mon[29435]: pgmap v607: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:11 smithi038 ceph-mon[32433]: pgmap v607: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:11 smithi186 ceph-mon[32774]: pgmap v607: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:13 smithi001 ceph-mon[29435]: pgmap v608: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:13.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:13 smithi038 ceph-mon[32433]: pgmap v608: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:13.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:13 smithi186 ceph-mon[32774]: pgmap v608: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:15 smithi001 ceph-mon[29435]: pgmap v609: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:15 smithi038 ceph-mon[32433]: pgmap v609: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:15 smithi186 ceph-mon[32774]: pgmap v609: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:17.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:17 smithi001 ceph-mon[29435]: pgmap v610: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:17.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:17 smithi038 ceph-mon[32433]: pgmap v610: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:17 smithi186 ceph-mon[32774]: pgmap v610: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:19.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:19 smithi001 ceph-mon[29435]: pgmap v611: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:19.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:19 smithi038 ceph-mon[32433]: pgmap v611: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:19.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:19 smithi186 ceph-mon[32774]: pgmap v611: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:21.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:21 smithi001 ceph-mon[29435]: pgmap v612: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:21.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:21 smithi038 ceph-mon[32433]: pgmap v612: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:21.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:21 smithi186 ceph-mon[32774]: pgmap v612: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:23.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:23 smithi001 ceph-mon[29435]: pgmap v613: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:23.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:23 smithi038 ceph-mon[32433]: pgmap v613: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:23 smithi186 ceph-mon[32774]: pgmap v613: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:25.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:25 smithi001 ceph-mon[29435]: pgmap v614: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:25.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:25 smithi038 ceph-mon[32433]: pgmap v614: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:25 smithi186 ceph-mon[32774]: pgmap v614: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:27.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:27 smithi001 ceph-mon[29435]: pgmap v615: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:27.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:27 smithi038 ceph-mon[32433]: pgmap v615: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:27 smithi186 ceph-mon[32774]: pgmap v615: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:29.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:29 smithi001 ceph-mon[29435]: pgmap v616: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:29.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:29 smithi038 ceph-mon[32433]: pgmap v616: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:29.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:29 smithi186 ceph-mon[32774]: pgmap v616: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:31 smithi001 ceph-mon[29435]: pgmap v617: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:31 smithi038 ceph-mon[32433]: pgmap v617: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:31 smithi186 ceph-mon[32774]: pgmap v617: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:33 smithi001 ceph-mon[29435]: pgmap v618: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:33 smithi038 ceph-mon[32433]: pgmap v618: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:33 smithi186 ceph-mon[32774]: pgmap v618: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:35 smithi001 ceph-mon[29435]: pgmap v619: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:35 smithi038 ceph-mon[32433]: pgmap v619: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:35 smithi186 ceph-mon[32774]: pgmap v619: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:37.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:37 smithi001 ceph-mon[29435]: pgmap v620: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:37 smithi038 ceph-mon[32433]: pgmap v620: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:37.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:37 smithi186 ceph-mon[32774]: pgmap v620: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:39.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:39 smithi001 ceph-mon[29435]: pgmap v621: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:39 smithi038 ceph-mon[32433]: pgmap v621: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:39.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:39 smithi186 ceph-mon[32774]: pgmap v621: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:41.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:41 smithi001 ceph-mon[29435]: pgmap v622: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:41.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:41 smithi038 ceph-mon[32433]: pgmap v622: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:41.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:41 smithi186 ceph-mon[32774]: pgmap v622: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:42.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:58:42.748+0000 7f39497fa640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f393c005ed0 con 0x7f3954104560 2024-04-26T22:58:42.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T22:58:42.748+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f393c005ed0 con 0x7f3954104560 2024-04-26T22:58:43.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:43 smithi001 ceph-mon[29435]: pgmap v623: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:43.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:43 smithi038 ceph-mon[32433]: pgmap v623: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:43 smithi186 ceph-mon[32774]: pgmap v623: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:45.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:45 smithi001 ceph-mon[29435]: pgmap v624: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:45.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:45 smithi038 ceph-mon[32433]: pgmap v624: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:45 smithi186 ceph-mon[32774]: pgmap v624: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:47.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:47 smithi001 ceph-mon[29435]: pgmap v625: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:47.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:47 smithi038 ceph-mon[32433]: pgmap v625: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:47 smithi186 ceph-mon[32774]: pgmap v625: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:49.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:49 smithi001 ceph-mon[29435]: pgmap v626: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:49.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:49 smithi038 ceph-mon[32433]: pgmap v626: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:49.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:49 smithi186 ceph-mon[32774]: pgmap v626: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:51.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:51 smithi001 ceph-mon[29435]: pgmap v627: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:51 smithi038 ceph-mon[32433]: pgmap v627: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:51.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:51 smithi186 ceph-mon[32774]: pgmap v627: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:53.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:53 smithi001 ceph-mon[29435]: pgmap v628: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:53 smithi038 ceph-mon[32433]: pgmap v628: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:53.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:53 smithi186 ceph-mon[32774]: pgmap v628: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:58:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:55 smithi001 ceph-mon[29435]: pgmap v629: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:55 smithi038 ceph-mon[32433]: pgmap v629: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:55 smithi186 ceph-mon[32774]: pgmap v629: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:58:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:57 smithi001 ceph-mon[29435]: pgmap v630: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:57.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:57 smithi038 ceph-mon[32433]: pgmap v630: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:57.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:57 smithi186 ceph-mon[32774]: pgmap v630: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:58:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:58:59 smithi001 ceph-mon[29435]: pgmap v631: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:58:59 smithi038 ceph-mon[32433]: pgmap v631: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:58:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:58:59 smithi186 ceph-mon[32774]: pgmap v631: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:01 smithi001 ceph-mon[29435]: pgmap v632: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:01 smithi038 ceph-mon[32433]: pgmap v632: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:01 smithi186 ceph-mon[32774]: pgmap v632: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:03 smithi038 ceph-mon[32433]: pgmap v633: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:03.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:03 smithi186 ceph-mon[32774]: pgmap v633: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:03.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:03 smithi001 ceph-mon[29435]: pgmap v633: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:05 smithi186 ceph-mon[32774]: pgmap v634: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:05.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:05 smithi001 ceph-mon[29435]: pgmap v634: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:05.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:05 smithi038 ceph-mon[32433]: pgmap v634: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:07 smithi186 ceph-mon[32774]: pgmap v635: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:07.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:07 smithi001 ceph-mon[29435]: pgmap v635: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:07.600 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:07 smithi038 ceph-mon[32433]: pgmap v635: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:09 smithi186 ceph-mon[32774]: pgmap v636: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:09.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:09 smithi001 ceph-mon[29435]: pgmap v636: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:09.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:09 smithi038 ceph-mon[32433]: pgmap v636: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:10.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:10 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:59:10.595 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:10 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:59:10.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:10 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:59:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:11 smithi186 ceph-mon[32774]: pgmap v637: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:11 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:59:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:11 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:59:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:11 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:59:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:11 smithi001 ceph-mon[29435]: pgmap v637: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:11 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:59:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:11 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:59:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:11 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:59:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:11 smithi038 ceph-mon[32433]: pgmap v637: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:11 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:59:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:11 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:59:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:11 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T22:59:13.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:13 smithi186 ceph-mon[32774]: pgmap v638: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:13 smithi001 ceph-mon[29435]: pgmap v638: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:13 smithi038 ceph-mon[32433]: pgmap v638: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:15 smithi186 ceph-mon[32774]: pgmap v639: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:15 smithi001 ceph-mon[29435]: pgmap v639: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:15 smithi038 ceph-mon[32433]: pgmap v639: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:17 smithi186 ceph-mon[32774]: pgmap v640: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:17 smithi001 ceph-mon[29435]: pgmap v640: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:17.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:17 smithi038 ceph-mon[32433]: pgmap v640: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:19.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:19 smithi186 ceph-mon[32774]: pgmap v641: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:19.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:19 smithi001 ceph-mon[29435]: pgmap v641: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:19 smithi038 ceph-mon[32433]: pgmap v641: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:21.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:21 smithi186 ceph-mon[32774]: pgmap v642: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:21 smithi001 ceph-mon[29435]: pgmap v642: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:21.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:21 smithi038 ceph-mon[32433]: pgmap v642: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:23.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:23 smithi001 ceph-mon[29435]: pgmap v643: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:23 smithi186 ceph-mon[32774]: pgmap v643: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:23.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:23 smithi038 ceph-mon[32433]: pgmap v643: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:25 smithi186 ceph-mon[32774]: pgmap v644: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:25 smithi001 ceph-mon[29435]: pgmap v644: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:25 smithi038 ceph-mon[32433]: pgmap v644: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:27.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:27 smithi038 ceph-mon[32433]: pgmap v645: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:27 smithi186 ceph-mon[32774]: pgmap v645: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:27 smithi001 ceph-mon[29435]: pgmap v645: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:29.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:29 smithi186 ceph-mon[32774]: pgmap v646: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:29 smithi001 ceph-mon[29435]: pgmap v646: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:29.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:29 smithi038 ceph-mon[32433]: pgmap v646: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:31 smithi186 ceph-mon[32774]: pgmap v647: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:31 smithi001 ceph-mon[29435]: pgmap v647: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:31 smithi038 ceph-mon[32433]: pgmap v647: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:33.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:32 smithi186 ceph-mon[32774]: pgmap v648: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:32 smithi001 ceph-mon[29435]: pgmap v648: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:32 smithi038 ceph-mon[32433]: pgmap v648: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:35 smithi001 ceph-mon[29435]: pgmap v649: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:35 smithi038 ceph-mon[32433]: pgmap v649: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:35 smithi186 ceph-mon[32774]: pgmap v649: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:37.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:37 smithi001 ceph-mon[29435]: pgmap v650: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:37 smithi038 ceph-mon[32433]: pgmap v650: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:37.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:37 smithi186 ceph-mon[32774]: pgmap v650: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:39.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:39 smithi001 ceph-mon[29435]: pgmap v651: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:39 smithi038 ceph-mon[32433]: pgmap v651: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:39.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:39 smithi186 ceph-mon[32774]: pgmap v651: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:41.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:41 smithi001 ceph-mon[29435]: pgmap v652: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:41.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:41 smithi038 ceph-mon[32433]: pgmap v652: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:41.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:41 smithi186 ceph-mon[32774]: pgmap v652: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:43.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:43 smithi001 ceph-mon[29435]: pgmap v653: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:43.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:43 smithi038 ceph-mon[32433]: pgmap v653: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:43 smithi186 ceph-mon[32774]: pgmap v653: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:45.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:45 smithi001 ceph-mon[29435]: pgmap v654: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:45.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:45 smithi038 ceph-mon[32433]: pgmap v654: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:45 smithi186 ceph-mon[32774]: pgmap v654: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:47.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:47 smithi001 ceph-mon[29435]: pgmap v655: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:47.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:47 smithi038 ceph-mon[32433]: pgmap v655: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:47 smithi186 ceph-mon[32774]: pgmap v655: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:49.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:49 smithi001 ceph-mon[29435]: pgmap v656: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:49.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:49 smithi038 ceph-mon[32433]: pgmap v656: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:49.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:49 smithi186 ceph-mon[32774]: pgmap v656: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:51.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:51 smithi001 ceph-mon[29435]: pgmap v657: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:51 smithi038 ceph-mon[32433]: pgmap v657: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:51.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:51 smithi186 ceph-mon[32774]: pgmap v657: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:53.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:53 smithi001 ceph-mon[29435]: pgmap v658: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:53 smithi038 ceph-mon[32433]: pgmap v658: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:53.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:53 smithi186 ceph-mon[32774]: pgmap v658: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T22:59:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:55 smithi001 ceph-mon[29435]: pgmap v659: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:55 smithi038 ceph-mon[32433]: pgmap v659: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:55 smithi186 ceph-mon[32774]: pgmap v659: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T22:59:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:57 smithi001 ceph-mon[29435]: pgmap v660: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:57.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:57 smithi038 ceph-mon[32433]: pgmap v660: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:57.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:57 smithi186 ceph-mon[32774]: pgmap v660: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T22:59:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 22:59:59 smithi001 ceph-mon[29435]: pgmap v661: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 22:59:59 smithi038 ceph-mon[32433]: pgmap v661: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T22:59:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 22:59:59 smithi186 ceph-mon[32774]: pgmap v661: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:00.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:00 smithi001 ceph-mon[29435]: overall HEALTH_OK 2024-04-26T23:00:00.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:00 smithi038 ceph-mon[32433]: overall HEALTH_OK 2024-04-26T23:00:00.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:00 smithi186 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T23:00:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:01 smithi001 ceph-mon[29435]: pgmap v662: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:01 smithi038 ceph-mon[32433]: pgmap v662: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:01 smithi186 ceph-mon[32774]: pgmap v662: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:03.235 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:02 smithi186 ceph-mon[32774]: pgmap v663: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:03.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:02 smithi001 ceph-mon[29435]: pgmap v663: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:02 smithi038 ceph-mon[32433]: pgmap v663: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:05.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:05 smithi001 ceph-mon[29435]: pgmap v664: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:05.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:05 smithi038 ceph-mon[32433]: pgmap v664: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:05 smithi186 ceph-mon[32774]: pgmap v664: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:07.345 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:07 smithi001 ceph-mon[29435]: pgmap v665: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:07 smithi038 ceph-mon[32433]: pgmap v665: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:07 smithi186 ceph-mon[32774]: pgmap v665: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:09.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:09 smithi001 ceph-mon[29435]: pgmap v666: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:09.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:09 smithi038 ceph-mon[32433]: pgmap v666: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:09 smithi186 ceph-mon[32774]: pgmap v666: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:11.303 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:11 smithi038 ceph-mon[32433]: pgmap v667: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:11.304 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:11 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:00:11.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:11 smithi001 ceph-mon[29435]: pgmap v667: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:11.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:11 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:00:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:11 smithi186 ceph-mon[32774]: pgmap v667: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:11 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:00:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: pgmap v668: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:13.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:00:13.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: pgmap v668: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:00:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: pgmap v668: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:00:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:00:15.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:15 smithi186 ceph-mon[32774]: pgmap v669: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:15 smithi001 ceph-mon[29435]: pgmap v669: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:15 smithi038 ceph-mon[32433]: pgmap v669: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:17.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:17 smithi186 ceph-mon[32774]: pgmap v670: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:17 smithi001 ceph-mon[29435]: pgmap v670: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:17 smithi038 ceph-mon[32433]: pgmap v670: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:19.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:19 smithi186 ceph-mon[32774]: pgmap v671: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:19 smithi001 ceph-mon[29435]: pgmap v671: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:19 smithi038 ceph-mon[32433]: pgmap v671: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:21.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:21 smithi186 ceph-mon[32774]: pgmap v672: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:21 smithi001 ceph-mon[29435]: pgmap v672: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:21 smithi038 ceph-mon[32433]: pgmap v672: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:23.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:23 smithi186 ceph-mon[32774]: pgmap v673: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:23 smithi001 ceph-mon[29435]: pgmap v673: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:23 smithi038 ceph-mon[32433]: pgmap v673: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:25.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:25 smithi186 ceph-mon[32774]: pgmap v674: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:25 smithi001 ceph-mon[29435]: pgmap v674: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:25 smithi038 ceph-mon[32433]: pgmap v674: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:27.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:27 smithi186 ceph-mon[32774]: pgmap v675: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:27 smithi001 ceph-mon[29435]: pgmap v675: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:27 smithi038 ceph-mon[32433]: pgmap v675: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:29.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:29 smithi186 ceph-mon[32774]: pgmap v676: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:29 smithi001 ceph-mon[29435]: pgmap v676: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:29 smithi038 ceph-mon[32433]: pgmap v676: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:31.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:31 smithi186 ceph-mon[32774]: pgmap v677: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:31 smithi001 ceph-mon[29435]: pgmap v677: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:31 smithi038 ceph-mon[32433]: pgmap v677: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:33.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:33 smithi186 ceph-mon[32774]: pgmap v678: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:33 smithi001 ceph-mon[29435]: pgmap v678: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:33.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:33 smithi038 ceph-mon[32433]: pgmap v678: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:35.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:35 smithi186 ceph-mon[32774]: pgmap v679: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:35 smithi001 ceph-mon[29435]: pgmap v679: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:35.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:35 smithi038 ceph-mon[32433]: pgmap v679: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:37.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:37 smithi186 ceph-mon[32774]: pgmap v680: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:37 smithi001 ceph-mon[29435]: pgmap v680: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:37.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:37 smithi038 ceph-mon[32433]: pgmap v680: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:39.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:39 smithi186 ceph-mon[32774]: pgmap v681: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:39.845 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:39 smithi001 ceph-mon[29435]: pgmap v681: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:39.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:39 smithi038 ceph-mon[32433]: pgmap v681: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:41.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:41 smithi186 ceph-mon[32774]: pgmap v682: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:41 smithi001 ceph-mon[29435]: pgmap v682: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:41 smithi038 ceph-mon[32433]: pgmap v682: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:43.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:43 smithi186 ceph-mon[32774]: pgmap v683: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:43 smithi001 ceph-mon[29435]: pgmap v683: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:43 smithi038 ceph-mon[32433]: pgmap v683: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:45.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:45 smithi186 ceph-mon[32774]: pgmap v684: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:45 smithi001 ceph-mon[29435]: pgmap v684: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:45 smithi038 ceph-mon[32433]: pgmap v684: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:47.737 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:47 smithi186 ceph-mon[32774]: pgmap v685: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:47 smithi001 ceph-mon[29435]: pgmap v685: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:47 smithi038 ceph-mon[32433]: pgmap v685: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:49.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:49 smithi186 ceph-mon[32774]: pgmap v686: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:49 smithi001 ceph-mon[29435]: pgmap v686: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:49 smithi038 ceph-mon[32433]: pgmap v686: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:51.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:51 smithi186 ceph-mon[32774]: pgmap v687: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:51 smithi001 ceph-mon[29435]: pgmap v687: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:51 smithi038 ceph-mon[32433]: pgmap v687: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:53.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:53 smithi186 ceph-mon[32774]: pgmap v688: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:53 smithi001 ceph-mon[29435]: pgmap v688: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:53 smithi038 ceph-mon[32433]: pgmap v688: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:00:55.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:55 smithi186 ceph-mon[32774]: pgmap v689: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:55 smithi001 ceph-mon[29435]: pgmap v689: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:55 smithi038 ceph-mon[32433]: pgmap v689: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:00:57.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:57 smithi186 ceph-mon[32774]: pgmap v690: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:57 smithi001 ceph-mon[29435]: pgmap v690: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:57 smithi038 ceph-mon[32433]: pgmap v690: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:00:59.735 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:00:59 smithi186 ceph-mon[32774]: pgmap v691: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:00:59 smithi001 ceph-mon[29435]: pgmap v691: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:00:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:00:59 smithi038 ceph-mon[32433]: pgmap v691: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:01.747 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:01 smithi186 ceph-mon[32774]: pgmap v692: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:01 smithi001 ceph-mon[29435]: pgmap v692: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:01 smithi038 ceph-mon[32433]: pgmap v692: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:03 smithi001 ceph-mon[29435]: pgmap v693: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:03 smithi038 ceph-mon[32433]: pgmap v693: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:03 smithi186 ceph-mon[32774]: pgmap v693: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:05 smithi001 ceph-mon[29435]: pgmap v694: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:05 smithi038 ceph-mon[32433]: pgmap v694: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:05 smithi186 ceph-mon[32774]: pgmap v694: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:07 smithi001 ceph-mon[29435]: pgmap v695: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:07 smithi038 ceph-mon[32433]: pgmap v695: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:07 smithi186 ceph-mon[32774]: pgmap v695: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:09 smithi001 ceph-mon[29435]: pgmap v696: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:09 smithi038 ceph-mon[32433]: pgmap v696: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:09 smithi186 ceph-mon[32774]: pgmap v696: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:11 smithi001 ceph-mon[29435]: pgmap v697: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:11 smithi038 ceph-mon[32433]: pgmap v697: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:11 smithi186 ceph-mon[32774]: pgmap v697: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:13 smithi001 ceph-mon[29435]: pgmap v698: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:13 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:01:13.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:13 smithi038 ceph-mon[32433]: pgmap v698: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:13 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:01:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:13 smithi186 ceph-mon[32774]: pgmap v698: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:13 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: pgmap v699: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:01:15.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:01:15.347 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:14 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: pgmap v699: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:01:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:01:15.352 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:14 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: pgmap v699: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:01:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:01:15.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:01:17.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:17 smithi001 ceph-mon[29435]: pgmap v700: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:17.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:17 smithi038 ceph-mon[32433]: pgmap v700: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:17 smithi186 ceph-mon[32774]: pgmap v700: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:19.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:19 smithi001 ceph-mon[29435]: pgmap v701: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:19.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:19 smithi038 ceph-mon[32433]: pgmap v701: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:19.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:19 smithi186 ceph-mon[32774]: pgmap v701: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:21.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:21 smithi001 ceph-mon[29435]: pgmap v702: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:21.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:21 smithi038 ceph-mon[32433]: pgmap v702: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:21.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:21 smithi186 ceph-mon[32774]: pgmap v702: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:23.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:23 smithi001 ceph-mon[29435]: pgmap v703: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:23.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:23 smithi038 ceph-mon[32433]: pgmap v703: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:23 smithi186 ceph-mon[32774]: pgmap v703: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:25.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:25 smithi001 ceph-mon[29435]: pgmap v704: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:25.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:25 smithi038 ceph-mon[32433]: pgmap v704: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:25 smithi186 ceph-mon[32774]: pgmap v704: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:27.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:27 smithi001 ceph-mon[29435]: pgmap v705: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:27.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:27 smithi038 ceph-mon[32433]: pgmap v705: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:27 smithi186 ceph-mon[32774]: pgmap v705: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:29.287 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:29 smithi186 ceph-mon[32774]: pgmap v706: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:29.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:29 smithi001 ceph-mon[29435]: pgmap v706: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:29.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:29 smithi038 ceph-mon[32433]: pgmap v706: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:31.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:31 smithi001 ceph-mon[29435]: pgmap v707: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:31.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:31 smithi038 ceph-mon[32433]: pgmap v707: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:31 smithi186 ceph-mon[32774]: pgmap v707: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:33 smithi001 ceph-mon[29435]: pgmap v708: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:33 smithi038 ceph-mon[32433]: pgmap v708: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:33 smithi186 ceph-mon[32774]: pgmap v708: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:35 smithi001 ceph-mon[29435]: pgmap v709: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:35 smithi038 ceph-mon[32433]: pgmap v709: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:35 smithi186 ceph-mon[32774]: pgmap v709: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:37.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:37 smithi001 ceph-mon[29435]: pgmap v710: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:37 smithi038 ceph-mon[32433]: pgmap v710: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:37.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:37 smithi186 ceph-mon[32774]: pgmap v710: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:39.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:39 smithi001 ceph-mon[29435]: pgmap v711: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:39 smithi038 ceph-mon[32433]: pgmap v711: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:39.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:39 smithi186 ceph-mon[32774]: pgmap v711: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:41.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:41 smithi001 ceph-mon[29435]: pgmap v712: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:41.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:41 smithi038 ceph-mon[32433]: pgmap v712: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:41.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:41 smithi186 ceph-mon[32774]: pgmap v712: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:01:42.773+0000 7f39497fa640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f393c0060f0 con 0x7f3954104560 2024-04-26T23:01:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:01:42.774+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f393c0060f0 con 0x7f3954104560 2024-04-26T23:01:43.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:43 smithi001 ceph-mon[29435]: pgmap v713: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:43.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:43 smithi038 ceph-mon[32433]: pgmap v713: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:43 smithi186 ceph-mon[32774]: pgmap v713: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:45.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:45 smithi001 ceph-mon[29435]: pgmap v714: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:45.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:45 smithi038 ceph-mon[32433]: pgmap v714: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:45 smithi186 ceph-mon[32774]: pgmap v714: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:47.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:47 smithi001 ceph-mon[29435]: pgmap v715: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:47.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:47 smithi038 ceph-mon[32433]: pgmap v715: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:47 smithi186 ceph-mon[32774]: pgmap v715: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:49.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:49 smithi001 ceph-mon[29435]: pgmap v716: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:49.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:49 smithi038 ceph-mon[32433]: pgmap v716: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:49.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:49 smithi186 ceph-mon[32774]: pgmap v716: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:51.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:51 smithi001 ceph-mon[29435]: pgmap v717: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:51 smithi038 ceph-mon[32433]: pgmap v717: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:51.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:51 smithi186 ceph-mon[32774]: pgmap v717: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:53.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:53 smithi001 ceph-mon[29435]: pgmap v718: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:53.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:53 smithi038 ceph-mon[32433]: pgmap v718: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:53.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:53 smithi186 ceph-mon[32774]: pgmap v718: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:01:55.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:55 smithi001 ceph-mon[29435]: pgmap v719: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:55.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:55 smithi038 ceph-mon[32433]: pgmap v719: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:55 smithi186 ceph-mon[32774]: pgmap v719: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:01:57.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:57 smithi001 ceph-mon[29435]: pgmap v720: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:57.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:57 smithi038 ceph-mon[32433]: pgmap v720: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:57.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:57 smithi186 ceph-mon[32774]: pgmap v720: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:01:59.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:01:59 smithi001 ceph-mon[29435]: pgmap v721: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:59.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:01:59 smithi038 ceph-mon[32433]: pgmap v721: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:01:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:01:59 smithi186 ceph-mon[32774]: pgmap v721: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:01.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:01 smithi001 ceph-mon[29435]: pgmap v722: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:01.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:01 smithi038 ceph-mon[32433]: pgmap v722: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:01.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:01 smithi186 ceph-mon[32774]: pgmap v722: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:03.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:03 smithi001 ceph-mon[29435]: pgmap v723: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:03.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:03 smithi038 ceph-mon[32433]: pgmap v723: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:03.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:03 smithi186 ceph-mon[32774]: pgmap v723: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:05.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:05 smithi001 ceph-mon[29435]: pgmap v724: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:05.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:05 smithi038 ceph-mon[32433]: pgmap v724: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:05 smithi186 ceph-mon[32774]: pgmap v724: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:07.345 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:07 smithi001 ceph-mon[29435]: pgmap v725: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:07.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:07 smithi038 ceph-mon[32433]: pgmap v725: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:07 smithi186 ceph-mon[32774]: pgmap v725: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:09.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:09 smithi001 ceph-mon[29435]: pgmap v726: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:09.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:09 smithi038 ceph-mon[32433]: pgmap v726: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:09 smithi186 ceph-mon[32774]: pgmap v726: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:11.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:11 smithi001 ceph-mon[29435]: pgmap v727: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:11.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:11 smithi038 ceph-mon[32433]: pgmap v727: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:11 smithi186 ceph-mon[32774]: pgmap v727: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:13.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:13 smithi001 ceph-mon[29435]: pgmap v728: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:13.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:13 smithi038 ceph-mon[32433]: pgmap v728: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:13.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:13 smithi186 ceph-mon[32774]: pgmap v728: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:15.345 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:15 smithi001 ceph-mon[29435]: pgmap v729: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:15.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:15 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:02:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:15 smithi038 ceph-mon[32433]: pgmap v729: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:15.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:15 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:02:15.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:15 smithi186 ceph-mon[32774]: pgmap v729: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:15 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:02:16.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:02:16.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:02:16.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:02:16.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:02:16.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:02:16.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:02:16.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:02:16.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:02:16.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:02:17.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:17 smithi001 ceph-mon[29435]: pgmap v730: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:17.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:17 smithi038 ceph-mon[32433]: pgmap v730: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:17.984 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:17 smithi186 ceph-mon[32774]: pgmap v730: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:19.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:19 smithi001 ceph-mon[29435]: pgmap v731: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:19.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:19 smithi038 ceph-mon[32433]: pgmap v731: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:19 smithi186 ceph-mon[32774]: pgmap v731: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:21.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:21 smithi001 ceph-mon[29435]: pgmap v732: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:21.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:21 smithi038 ceph-mon[32433]: pgmap v732: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:21 smithi186 ceph-mon[32774]: pgmap v732: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:23.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:23 smithi001 ceph-mon[29435]: pgmap v733: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:23.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:23 smithi038 ceph-mon[32433]: pgmap v733: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:23 smithi186 ceph-mon[32774]: pgmap v733: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:25.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:25 smithi001 ceph-mon[29435]: pgmap v734: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:25.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:25 smithi038 ceph-mon[32433]: pgmap v734: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:25 smithi186 ceph-mon[32774]: pgmap v734: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:27.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:27 smithi001 ceph-mon[29435]: pgmap v735: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:27.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:27 smithi038 ceph-mon[32433]: pgmap v735: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:27 smithi186 ceph-mon[32774]: pgmap v735: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:29.786 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:29 smithi186 ceph-mon[32774]: pgmap v736: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:29.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:29 smithi001 ceph-mon[29435]: pgmap v736: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:29.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:29 smithi038 ceph-mon[32433]: pgmap v736: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:31.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:31 smithi001 ceph-mon[29435]: pgmap v737: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:31.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:31 smithi038 ceph-mon[32433]: pgmap v737: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:31.984 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:31 smithi186 ceph-mon[32774]: pgmap v737: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:33.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:33 smithi001 ceph-mon[29435]: pgmap v738: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:33.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:33 smithi038 ceph-mon[32433]: pgmap v738: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:33.984 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:33 smithi186 ceph-mon[32774]: pgmap v738: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:35.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:35 smithi001 ceph-mon[29435]: pgmap v739: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:35.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:35 smithi038 ceph-mon[32433]: pgmap v739: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:35.984 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:35 smithi186 ceph-mon[32774]: pgmap v739: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:37.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:37 smithi001 ceph-mon[29435]: pgmap v740: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:37.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:37 smithi038 ceph-mon[32433]: pgmap v740: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:37.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:37 smithi186 ceph-mon[32774]: pgmap v740: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:39.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:39 smithi001 ceph-mon[29435]: pgmap v741: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:39.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:39 smithi038 ceph-mon[32433]: pgmap v741: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:39.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:39 smithi186 ceph-mon[32774]: pgmap v741: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:41.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:41 smithi001 ceph-mon[29435]: pgmap v742: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:41.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:41 smithi038 ceph-mon[32433]: pgmap v742: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:41.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:41 smithi186 ceph-mon[32774]: pgmap v742: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:43.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:43 smithi001 ceph-mon[29435]: pgmap v743: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:43.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:43 smithi038 ceph-mon[32433]: pgmap v743: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:43.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:43 smithi186 ceph-mon[32774]: pgmap v743: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:45.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:45 smithi001 ceph-mon[29435]: pgmap v744: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:45.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:45 smithi038 ceph-mon[32433]: pgmap v744: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:45.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:45 smithi186 ceph-mon[32774]: pgmap v744: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:47.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:47 smithi001 ceph-mon[29435]: pgmap v745: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:47.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:47 smithi038 ceph-mon[32433]: pgmap v745: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:47.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:47 smithi186 ceph-mon[32774]: pgmap v745: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:49.845 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:49 smithi001 ceph-mon[29435]: pgmap v746: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:49.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:49 smithi038 ceph-mon[32433]: pgmap v746: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:49 smithi186 ceph-mon[32774]: pgmap v746: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:51.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:51 smithi001 ceph-mon[29435]: pgmap v747: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:51.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:51 smithi038 ceph-mon[32433]: pgmap v747: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:51.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:51 smithi186 ceph-mon[32774]: pgmap v747: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:02:53.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:53 smithi001 ceph-mon[29435]: pgmap v748: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:53.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:53 smithi038 ceph-mon[32433]: pgmap v748: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:53.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:53 smithi186 ceph-mon[32774]: pgmap v748: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:02:55.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:55 smithi001 ceph-mon[29435]: pgmap v749: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:55.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:55 smithi038 ceph-mon[32433]: pgmap v749: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:55.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:55 smithi186 ceph-mon[32774]: pgmap v749: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:02:57.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:57 smithi001 ceph-mon[29435]: pgmap v750: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:57.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:57 smithi038 ceph-mon[32433]: pgmap v750: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:57.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:57 smithi186 ceph-mon[32774]: pgmap v750: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:02:59.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:02:59 smithi001 ceph-mon[29435]: pgmap v751: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:59.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:02:59 smithi038 ceph-mon[32433]: pgmap v751: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:02:59.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:02:59 smithi186 ceph-mon[32774]: pgmap v751: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:01.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:01 smithi001 ceph-mon[29435]: pgmap v752: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:01.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:01 smithi038 ceph-mon[32433]: pgmap v752: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:01.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:01 smithi186 ceph-mon[32774]: pgmap v752: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:03.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:03 smithi001 ceph-mon[29435]: pgmap v753: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:03.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:03 smithi038 ceph-mon[32433]: pgmap v753: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:03.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:03 smithi186 ceph-mon[32774]: pgmap v753: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:05.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:05 smithi001 ceph-mon[29435]: pgmap v754: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:05.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:05 smithi038 ceph-mon[32433]: pgmap v754: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:05.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:05 smithi186 ceph-mon[32774]: pgmap v754: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:07.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:07 smithi001 ceph-mon[29435]: pgmap v755: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:07.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:07 smithi038 ceph-mon[32433]: pgmap v755: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:07.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:07 smithi186 ceph-mon[32774]: pgmap v755: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:09.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:09 smithi001 ceph-mon[29435]: pgmap v756: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:09.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:09 smithi038 ceph-mon[32433]: pgmap v756: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:09.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:09 smithi186 ceph-mon[32774]: pgmap v756: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:11.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:11 smithi001 ceph-mon[29435]: pgmap v757: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:11.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:11 smithi038 ceph-mon[32433]: pgmap v757: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:11.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:11 smithi186 ceph-mon[32774]: pgmap v757: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:13.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:13 smithi001 ceph-mon[29435]: pgmap v758: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:13.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:13 smithi038 ceph-mon[32433]: pgmap v758: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:13.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:13 smithi186 ceph-mon[32774]: pgmap v758: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:15.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:15 smithi001 ceph-mon[29435]: pgmap v759: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:15.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:15 smithi038 ceph-mon[32433]: pgmap v759: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:15.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:15 smithi186 ceph-mon[32774]: pgmap v759: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:16.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:03:16.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:03:16.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:03:16.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:16 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:03:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:03:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:03:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:03:16.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:16 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:03:17.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:03:17.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:03:17.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:03:17.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:16 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:03:17.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:17 smithi186 ceph-mon[32774]: pgmap v760: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:18.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:17 smithi001 ceph-mon[29435]: pgmap v760: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:18.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:17 smithi038 ceph-mon[32433]: pgmap v760: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:19.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:19 smithi186 ceph-mon[32774]: pgmap v761: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:20.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:19 smithi001 ceph-mon[29435]: pgmap v761: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:20.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:19 smithi038 ceph-mon[32433]: pgmap v761: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:21.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:21 smithi186 ceph-mon[32774]: pgmap v762: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:22.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:21 smithi001 ceph-mon[29435]: pgmap v762: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:22.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:21 smithi038 ceph-mon[32433]: pgmap v762: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:23.898 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:23 smithi001 ceph-mon[29435]: pgmap v763: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:23.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:23 smithi186 ceph-mon[32774]: pgmap v763: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:24.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:23 smithi038 ceph-mon[32433]: pgmap v763: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:25.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:25 smithi186 ceph-mon[32774]: pgmap v764: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:26.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:25 smithi001 ceph-mon[29435]: pgmap v764: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:26.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:25 smithi038 ceph-mon[32433]: pgmap v764: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:27.903 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:27 smithi038 ceph-mon[32433]: pgmap v765: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:27.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:27 smithi186 ceph-mon[32774]: pgmap v765: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:28.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:27 smithi001 ceph-mon[29435]: pgmap v765: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:29.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:29 smithi186 ceph-mon[32774]: pgmap v766: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:30.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:29 smithi001 ceph-mon[29435]: pgmap v766: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:30.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:29 smithi038 ceph-mon[32433]: pgmap v766: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:31.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:31 smithi186 ceph-mon[32774]: pgmap v767: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:32.096 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:31 smithi001 ceph-mon[29435]: pgmap v767: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:32.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:31 smithi038 ceph-mon[32433]: pgmap v767: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:33.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:32 smithi001 ceph-mon[29435]: pgmap v768: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:33.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:32 smithi038 ceph-mon[32433]: pgmap v768: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:32 smithi186 ceph-mon[32774]: pgmap v768: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:35.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:35 smithi001 ceph-mon[29435]: pgmap v769: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:35.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:35 smithi038 ceph-mon[32433]: pgmap v769: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:35 smithi186 ceph-mon[32774]: pgmap v769: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:37.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:37 smithi038 ceph-mon[32433]: pgmap v770: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:37.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:37 smithi186 ceph-mon[32774]: pgmap v770: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:37.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:37 smithi001 ceph-mon[29435]: pgmap v770: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:39.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:39 smithi038 ceph-mon[32433]: pgmap v771: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:39.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:39 smithi186 ceph-mon[32774]: pgmap v771: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:39.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:39 smithi001 ceph-mon[29435]: pgmap v771: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:41.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:41 smithi186 ceph-mon[32774]: pgmap v772: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:41.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:41 smithi001 ceph-mon[29435]: pgmap v772: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:41.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:41 smithi038 ceph-mon[32433]: pgmap v772: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:43 smithi186 ceph-mon[32774]: pgmap v773: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:43.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:43 smithi001 ceph-mon[29435]: pgmap v773: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:43.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:43 smithi038 ceph-mon[32433]: pgmap v773: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:45.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:45 smithi186 ceph-mon[32774]: pgmap v774: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:45.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:45 smithi001 ceph-mon[29435]: pgmap v774: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:45 smithi038 ceph-mon[32433]: pgmap v774: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:47 smithi186 ceph-mon[32774]: pgmap v775: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:47.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:47 smithi001 ceph-mon[29435]: pgmap v775: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:47 smithi038 ceph-mon[32433]: pgmap v775: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:49.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:49 smithi186 ceph-mon[32774]: pgmap v776: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:49.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:49 smithi001 ceph-mon[29435]: pgmap v776: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:49.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:49 smithi038 ceph-mon[32433]: pgmap v776: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:51.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:51 smithi186 ceph-mon[32774]: pgmap v777: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:51.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:51 smithi001 ceph-mon[29435]: pgmap v777: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:51.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:51 smithi038 ceph-mon[32433]: pgmap v777: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:03:53.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:53 smithi001 ceph-mon[29435]: pgmap v778: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:53.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:53 smithi186 ceph-mon[32774]: pgmap v778: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:53.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:53 smithi038 ceph-mon[32433]: pgmap v778: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:03:55.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:55 smithi186 ceph-mon[32774]: pgmap v779: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:55.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:55 smithi001 ceph-mon[29435]: pgmap v779: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:55.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:55 smithi038 ceph-mon[32433]: pgmap v779: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:03:57.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:57 smithi038 ceph-mon[32433]: pgmap v780: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:57.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:57 smithi186 ceph-mon[32774]: pgmap v780: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:57.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:57 smithi001 ceph-mon[29435]: pgmap v780: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:03:59.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:03:59 smithi186 ceph-mon[32774]: pgmap v781: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:59.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:03:59 smithi001 ceph-mon[29435]: pgmap v781: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:03:59.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:03:59 smithi038 ceph-mon[32433]: pgmap v781: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:01.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:01 smithi186 ceph-mon[32774]: pgmap v782: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:01.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:01 smithi001 ceph-mon[29435]: pgmap v782: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:01.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:01 smithi038 ceph-mon[32433]: pgmap v782: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:03.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:03 smithi186 ceph-mon[32774]: pgmap v783: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:03.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:03 smithi001 ceph-mon[29435]: pgmap v783: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:03.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:03 smithi038 ceph-mon[32433]: pgmap v783: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:05.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:05 smithi186 ceph-mon[32774]: pgmap v784: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:05.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:05 smithi001 ceph-mon[29435]: pgmap v784: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:05.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:05 smithi038 ceph-mon[32433]: pgmap v784: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:07.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:07 smithi186 ceph-mon[32774]: pgmap v785: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:07.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:07 smithi001 ceph-mon[29435]: pgmap v785: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:07.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:07 smithi038 ceph-mon[32433]: pgmap v785: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:09.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:09 smithi186 ceph-mon[32774]: pgmap v786: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:09.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:09 smithi001 ceph-mon[29435]: pgmap v786: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:09.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:09 smithi038 ceph-mon[32433]: pgmap v786: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:11.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:11 smithi186 ceph-mon[32774]: pgmap v787: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:11.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:11 smithi001 ceph-mon[29435]: pgmap v787: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:11.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:11 smithi038 ceph-mon[32433]: pgmap v787: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:13.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:13 smithi186 ceph-mon[32774]: pgmap v788: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:13.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:13 smithi001 ceph-mon[29435]: pgmap v788: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:13.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:13 smithi038 ceph-mon[32433]: pgmap v788: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:15.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:15 smithi186 ceph-mon[32774]: pgmap v789: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:15.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:15 smithi001 ceph-mon[29435]: pgmap v789: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:15.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:15 smithi038 ceph-mon[32433]: pgmap v789: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:17 smithi186 ceph-mon[32774]: pgmap v790: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:17 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:04:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:17 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:04:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:17 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:04:17.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:17 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:04:17.595 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:17 smithi001 ceph-mon[29435]: pgmap v790: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:17 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:04:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:17 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:04:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:17 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:04:17.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:17 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:04:17.600 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:17 smithi038 ceph-mon[32433]: pgmap v790: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:17.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:17 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:04:17.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:17 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:04:17.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:17 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:04:17.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:17 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' 2024-04-26T23:04:19.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:19 smithi186 ceph-mon[32774]: pgmap v791: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:19.595 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:19 smithi001 ceph-mon[29435]: pgmap v791: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:19.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:19 smithi038 ceph-mon[32433]: pgmap v791: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:21.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:21 smithi186 ceph-mon[32774]: pgmap v792: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:21.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:21 smithi001 ceph-mon[29435]: pgmap v792: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:21.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:21 smithi038 ceph-mon[32433]: pgmap v792: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:23.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:23 smithi186 ceph-mon[32774]: pgmap v793: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:23.595 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:23 smithi001 ceph-mon[29435]: pgmap v793: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:23.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:23 smithi038 ceph-mon[32433]: pgmap v793: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:25.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:25 smithi186 ceph-mon[32774]: pgmap v794: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:25.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:25 smithi001 ceph-mon[29435]: pgmap v794: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:25.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:25 smithi038 ceph-mon[32433]: pgmap v794: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:27.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:27 smithi186 ceph-mon[32774]: pgmap v795: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:27.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:27 smithi001 ceph-mon[29435]: pgmap v795: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:27.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:27 smithi038 ceph-mon[32433]: pgmap v795: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:29.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:29 smithi186 ceph-mon[32774]: pgmap v796: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:29.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:29 smithi001 ceph-mon[29435]: pgmap v796: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:29.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:29 smithi038 ceph-mon[32433]: pgmap v796: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:31.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:31 smithi186 ceph-mon[32774]: pgmap v797: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:31.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:31 smithi001 ceph-mon[29435]: pgmap v797: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:31.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:31 smithi038 ceph-mon[32433]: pgmap v797: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:33.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:33 smithi186 ceph-mon[32774]: pgmap v798: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:33.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:33 smithi001 ceph-mon[29435]: pgmap v798: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:33.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:33 smithi038 ceph-mon[32433]: pgmap v798: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:35.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:35 smithi186 ceph-mon[32774]: pgmap v799: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:35.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:35 smithi001 ceph-mon[29435]: pgmap v799: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:35.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:35 smithi038 ceph-mon[32433]: pgmap v799: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:37.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:37 smithi186 ceph-mon[32774]: pgmap v800: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:37.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:37 smithi001 ceph-mon[29435]: pgmap v800: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:37.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:37 smithi038 ceph-mon[32433]: pgmap v800: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T23:04:39.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:39 smithi186 ceph-mon[32774]: pgmap v801: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:39.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:39 smithi001 ceph-mon[29435]: pgmap v801: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:39.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:39 smithi038 ceph-mon[32433]: pgmap v801: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T23:04:41.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:41 smithi186 ceph-mon[32774]: pgmap v802: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:41.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:41 smithi001 ceph-mon[29435]: pgmap v802: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:41.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:41 smithi038 ceph-mon[32433]: pgmap v802: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.700+0000 7f395904d640 1 -- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 msgr2=0x7f39400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900024787) for more than 900000000 us, fault. 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.700+0000 7f395904d640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 0x7f39400a60f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3950005fd0 tx=0x7f395000b390 comp rx=0 tx=0).stop 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.700+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 msgr2=0x7f39400a60f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.700+0000 7f394a7fc640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400a3c30 0x7f39400a60f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.700+0000 7f394a7fc640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400ada90 0x7f39400afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.701+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f394c03de50 con 0x7f39400ada90 2024-04-26T23:04:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.701+0000 7f395904d640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400ada90 0x7f39400afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.701+0000 7f395904d640 1 --2- 172.21.15.1:0/3308630788 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f39400ada90 0x7f39400afe80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f395000b3d0 tx=0x7f3950016000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.799+0000 7f39497fa640 1 -- 172.21.15.1:0/3308630788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f393c006330 con 0x7f3954104560 2024-04-26T23:04:42.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:42.799+0000 7f394a7fc640 1 -- 172.21.15.1:0/3308630788 <== mon.0 v2:172.21.15.1:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f393c006330 con 0x7f3954104560 2024-04-26T23:04:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-26T23:04:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-26T23:04:42.899 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T23:04:43.168 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=24534ae538224d33b4799328ad369c94 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T23:04:43.170 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T23:04:43.179 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T23:04:43.180 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T23:04:43.180 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:43.204 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T23:04:43.204 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:43.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 -- 172.21.15.1:0/3840589040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50981034b0 msgr2=0x7f50981058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 --2- 172.21.15.1:0/3840589040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50981034b0 0x7f50981058a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5094009900 tx=0x7f509402a410 comp rx=0 tx=0).stop 2024-04-26T23:04:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 -- 172.21.15.1:0/3840589040 shutdown_connections 2024-04-26T23:04:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 --2- 172.21.15.1:0/3840589040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5098108710 0x7f509810ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 --2- 172.21.15.1:0/3840589040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5098105de0 0x7f50981081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 --2- 172.21.15.1:0/3840589040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50981034b0 0x7f50981058a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 -- 172.21.15.1:0/3840589040 >> 172.21.15.1:0/3840589040 conn(0x7f50980fce00 msgr2=0x7f50980ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 -- 172.21.15.1:0/3840589040 shutdown_connections 2024-04-26T23:04:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.420+0000 7f50a066d640 1 -- 172.21.15.1:0/3840589040 wait complete. 2024-04-26T23:04:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 Processor -- start 2024-04-26T23:04:43.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 -- start start 2024-04-26T23:04:43.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50981034b0 0x7f5098198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 0x7f5098199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5098108710 0x7f50981a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f509810e4b0 con 0x7f5098105de0 2024-04-26T23:04:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f509810e330 con 0x7f5098108710 2024-04-26T23:04:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.421+0000 7f50a066d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f509810e630 con 0x7f50981034b0 2024-04-26T23:04:43.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 0x7f5098199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 0x7f5098199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52848/0 (socket says 172.21.15.1:52848) 2024-04-26T23:04:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 -- 172.21.15.1:0/3551580454 learned_addr learned my addr 172.21.15.1:0/3551580454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509e409640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50981034b0 0x7f5098198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 -- 172.21.15.1:0/3551580454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50981034b0 msgr2=0x7f5098198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50981034b0 0x7f5098198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 -- 172.21.15.1:0/3551580454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5098108710 msgr2=0x7f50981a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509ec0a640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5098108710 0x7f50981a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5098108710 0x7f50981a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 -- 172.21.15.1:0/3551580454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50981a0800 con 0x7f5098105de0 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f509dc08640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 0x7f5098199120 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5088005fd0 tx=0x7f508800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088017070 con 0x7f5098105de0 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5088009c30 con 0x7f5098105de0 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088003d30 con 0x7f5098105de0 2024-04-26T23:04:43.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50981a0af0 con 0x7f5098105de0 2024-04-26T23:04:43.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.422+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50981a1090 con 0x7f5098105de0 2024-04-26T23:04:43.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.423+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50980fe8f0 con 0x7f5098105de0 2024-04-26T23:04:43.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.425+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5088003ed0 con 0x7f5098105de0 2024-04-26T23:04:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.425+0000 7f508f7fe640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f50840a3cc0 0x7f50840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.425+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f50880d0a00 con 0x7f5098105de0 2024-04-26T23:04:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.425+0000 7f509e409640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f50840a3cc0 0x7f50840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.426+0000 7f509e409640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f50840a3cc0 0x7f50840a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5094000c00 tx=0x7f50940314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.429+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5088020030 con 0x7f5098105de0 2024-04-26T23:04:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:43 smithi186 ceph-mon[32774]: pgmap v803: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:43.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:43 smithi186 ceph-mon[32774]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:04:43.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:43 smithi001 ceph-mon[29435]: pgmap v803: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:43.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:43 smithi001 ceph-mon[29435]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:04:43.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:43 smithi038 ceph-mon[32433]: pgmap v803: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-26T23:04:43.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:43 smithi038 ceph-mon[32433]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:04:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.617+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f509802dcd0 con 0x7f5098105de0 2024-04-26T23:04:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.618+0000 7f508f7fe640 1 -- 172.21.15.1:0/3551580454 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7f5088009de0 con 0x7f5098105de0 2024-04-26T23:04:43.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T23:04:43.619 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f50840a3cc0 msgr2=0x7f50840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f50840a3cc0 0x7f50840a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5094000c00 tx=0x7f50940314a0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 msgr2=0x7f5098199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 0x7f5098199120 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5088005fd0 tx=0x7f508800dde0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 shutdown_connections 2024-04-26T23:04:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f50840a3cc0 0x7f50840a6180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5098108710 0x7f50981a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5098105de0 0x7f5098199120 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 --2- 172.21.15.1:0/3551580454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50981034b0 0x7f5098198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 >> 172.21.15.1:0/3551580454 conn(0x7f50980fce00 msgr2=0x7f5098103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 shutdown_connections 2024-04-26T23:04:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.621+0000 7f50a066d640 1 -- 172.21.15.1:0/3551580454 wait complete. 2024-04-26T23:04:43.632 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T22:42:55.759310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T22:47:39.631594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T22:47:40.081969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:04:43.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:04:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 -- 172.21.15.1:0/1233698468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4102a70 msgr2=0x7fe1d4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/1233698468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4102a70 0x7fe1d4102ef0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009900 tx=0x7fe1c402a410 comp rx=0 tx=0).stop 2024-04-26T23:04:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 -- 172.21.15.1:0/1233698468 shutdown_connections 2024-04-26T23:04:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/1233698468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d4103430 0x7fe1d4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/1233698468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4102a70 0x7fe1d4102ef0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/1233698468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4101880 0x7fe1d4101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 -- 172.21.15.1:0/1233698468 >> 172.21.15.1:0/1233698468 conn(0x7fe1d40fce10 msgr2=0x7fe1d40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 -- 172.21.15.1:0/1233698468 shutdown_connections 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 -- 172.21.15.1:0/1233698468 wait complete. 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 Processor -- start 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.738+0000 7fe1db5ec640 1 -- start start 2024-04-26T23:04:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 0x7fe1d41057c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d4102a70 0x7fe1d4103e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4103430 0x7fe1d4104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410d3b0 con 0x7fe1d4101880 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410d230 con 0x7fe1d4103430 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410d530 con 0x7fe1d4102a70 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 0x7fe1d41057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 0x7fe1d41057c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52860/0 (socket says 172.21.15.1:52860) 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 -- 172.21.15.1:0/263617070 learned_addr learned my addr 172.21.15.1:0/263617070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:43.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d8b87640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d4102a70 0x7fe1d4103e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9b89640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4103430 0x7fe1d4104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 -- 172.21.15.1:0/263617070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d4102a70 msgr2=0x7fe1d4103e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d4102a70 0x7fe1d4103e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 -- 172.21.15.1:0/263617070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4103430 msgr2=0x7fe1d4104390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4103430 0x7fe1d4104390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 -- 172.21.15.1:0/263617070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d4104c50 con 0x7fe1d4101880 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9b89640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4103430 0x7fe1d4104390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:43.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1d9388640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 0x7fe1d41057c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000e9c0 tx=0x7fe1d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d000cd80 con 0x7fe1d4101880 2024-04-26T23:04:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d00035f0 con 0x7fe1d4101880 2024-04-26T23:04:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d41a2d50 con 0x7fe1d4101880 2024-04-26T23:04:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0010690 con 0x7fe1d4101880 2024-04-26T23:04:43.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.739+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d41a3200 con 0x7fe1d4101880 2024-04-26T23:04:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.740+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d41a3490 con 0x7fe1d4101880 2024-04-26T23:04:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.742+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fe1d00037a0 con 0x7fe1d4101880 2024-04-26T23:04:43.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.742+0000 7fe1ca7fc640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe1c00a3cc0 0x7fe1c00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.743+0000 7fe1d8b87640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe1c00a3cc0 0x7fe1c00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.743+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fe1d00d0a50 con 0x7fe1d4101880 2024-04-26T23:04:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.743+0000 7fe1d8b87640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe1c00a3cc0 0x7fe1c00a6180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4105530 tx=0x7fe1c40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:43.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.746+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe1d009d300 con 0x7fe1d4101880 2024-04-26T23:04:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.931+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1d41a37d0 con 0x7fe1d4101880 2024-04-26T23:04:43.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.931+0000 7fe1ca7fc640 1 -- 172.21.15.1:0/263617070 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fe1d009cc00 con 0x7fe1d4101880 2024-04-26T23:04:43.931 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T23:04:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe1c00a3cc0 msgr2=0x7fe1c00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe1c00a3cc0 0x7fe1c00a6180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4105530 tx=0x7fe1c40314a0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 msgr2=0x7fe1d41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 0x7fe1d41057c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000e9c0 tx=0x7fe1d000ee80 comp rx=0 tx=0).stop 2024-04-26T23:04:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 shutdown_connections 2024-04-26T23:04:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fe1c00a3cc0 0x7fe1c00a6180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d4102a70 0x7fe1d4103e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d4103430 0x7fe1d4104390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 --2- 172.21.15.1:0/263617070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1d4101880 0x7fe1d41057c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:43.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 >> 172.21.15.1:0/263617070 conn(0x7fe1d40fce10 msgr2=0x7fe1d40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:43.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 shutdown_connections 2024-04-26T23:04:43.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:43.933+0000 7fe1db5ec640 1 -- 172.21.15.1:0/263617070 wait complete. 2024-04-26T23:04:43.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:04:44.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 -- 172.21.15.1:0/884213994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 msgr2=0x7fc9f4101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/884213994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4101ca0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009900 tx=0x7fc9f002a410 comp rx=0 tx=0).stop 2024-04-26T23:04:44.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 -- 172.21.15.1:0/884213994 shutdown_connections 2024-04-26T23:04:44.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/884213994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4103430 0x7fc9f4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/884213994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9f4102a70 0x7fc9f4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/884213994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4101ca0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 -- 172.21.15.1:0/884213994 >> 172.21.15.1:0/884213994 conn(0x7fc9f40fce10 msgr2=0x7fc9f40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 -- 172.21.15.1:0/884213994 shutdown_connections 2024-04-26T23:04:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.049+0000 7fc9fb02a640 1 -- 172.21.15.1:0/884213994 wait complete. 2024-04-26T23:04:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 Processor -- start 2024-04-26T23:04:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 -- start start 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4102a70 0x7fc9f4199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9f4103430 0x7fc9f41a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f410d380 con 0x7fc9f4101880 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f410d200 con 0x7fc9f4103430 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9fb02a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f410d500 con 0x7fc9f4102a70 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52872/0 (socket says 172.21.15.1:52872) 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 -- 172.21.15.1:0/1438589913 learned_addr learned my addr 172.21.15.1:0/1438589913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f95c7640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9f4103430 0x7fc9f41a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9ebfff640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4102a70 0x7fc9f4199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 -- 172.21.15.1:0/1438589913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4102a70 msgr2=0x7fc9f4199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4102a70 0x7fc9f4199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 -- 172.21.15.1:0/1438589913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9f4103430 msgr2=0x7fc9f41a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9f4103430 0x7fc9f41a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9f8dc6640 1 -- 172.21.15.1:0/1438589913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f41a0860 con 0x7fc9f4101880 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.050+0000 7fc9ebfff640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4102a70 0x7fc9f4199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.051+0000 7fc9f8dc6640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4198c50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0006010 tx=0x7fc9f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.051+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f003f030 con 0x7fc9f4101880 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.051+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9f00318f0 con 0x7fc9f4101880 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.051+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f41a0af0 con 0x7fc9f4101880 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.051+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0035070 con 0x7fc9f4101880 2024-04-26T23:04:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.051+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f41a1030 con 0x7fc9f4101880 2024-04-26T23:04:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.052+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f41a1200 con 0x7fc9f4101880 2024-04-26T23:04:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.053+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc9f0042030 con 0x7fc9f4101880 2024-04-26T23:04:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.054+0000 7fc9e9ffb640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc9dc0a3cc0 0x7fc9dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.054+0000 7fc9ebfff640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc9dc0a3cc0 0x7fc9dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.054+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fc9f00f0f10 con 0x7fc9f4101880 2024-04-26T23:04:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.055+0000 7fc9ebfff640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc9dc0a3cc0 0x7fc9dc0a6180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40058c0 tx=0x7fc9e4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.058+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc9f00bd7c0 con 0x7fc9f4101880 2024-04-26T23:04:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.245+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc9f41a13d0 con 0x7fc9f4101880 2024-04-26T23:04:44.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.246+0000 7fc9e9ffb640 1 -- 172.21.15.1:0/1438589913 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fc9f00bd0c0 con 0x7fc9f4101880 2024-04-26T23:04:44.246 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T23:04:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc9dc0a3cc0 msgr2=0x7fc9dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc9dc0a3cc0 0x7fc9dc0a6180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40058c0 tx=0x7fc9e4005850 comp rx=0 tx=0).stop 2024-04-26T23:04:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 msgr2=0x7fc9f4198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4198c50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0006010 tx=0x7fc9f00042f0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 shutdown_connections 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fc9dc0a3cc0 0x7fc9dc0a6180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f4102a70 0x7fc9f4199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9f4103430 0x7fc9f41a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 --2- 172.21.15.1:0/1438589913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4101880 0x7fc9f4198c50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 >> 172.21.15.1:0/1438589913 conn(0x7fc9f40fce10 msgr2=0x7fc9f406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 shutdown_connections 2024-04-26T23:04:44.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.248+0000 7fc9fb02a640 1 -- 172.21.15.1:0/1438589913 wait complete. 2024-04-26T23:04:44.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:04:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/3834767239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08105d20 msgr2=0x7fcc08108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/3834767239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08105d20 0x7fcc08108110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009900 tx=0x7fcbf002a410 comp rx=0 tx=0).stop 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/3834767239 shutdown_connections 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/3834767239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc08108650 0x7fcc0810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/3834767239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08105d20 0x7fcc08108110 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/3834767239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc081033f0 0x7fcc081057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/3834767239 >> 172.21.15.1:0/3834767239 conn(0x7fcc080fce00 msgr2=0x7fcc080ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/3834767239 shutdown_connections 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.363+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/3834767239 wait complete. 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 Processor -- start 2024-04-26T23:04:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 -- start start 2024-04-26T23:04:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc081033f0 0x7fcc08198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc08105d20 0x7fcc08199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 0x7fcc081a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0810e490 con 0x7fcc08108650 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0810e310 con 0x7fcc081033f0 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.364+0000 7fcc0d1a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0810e610 con 0x7fcc08105d20 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 0x7fcc081a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 0x7fcc081a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52892/0 (socket says 172.21.15.1:52892) 2024-04-26T23:04:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 -- 172.21.15.1:0/2459151608 learned_addr learned my addr 172.21.15.1:0/2459151608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0659c640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc08105d20 0x7fcc08199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc06d9d640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc081033f0 0x7fcc08198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 -- 172.21.15.1:0/2459151608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc08105d20 msgr2=0x7fcc08199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc08105d20 0x7fcc08199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 -- 172.21.15.1:0/2459151608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc081033f0 msgr2=0x7fcc08198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc081033f0 0x7fcc08198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 -- 172.21.15.1:0/2459151608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc081a0840 con 0x7fcc08108650 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc06d9d640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc081033f0 0x7fcc08198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0759e640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 0x7fcc081a01a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800eaa0 tx=0x7fcbf800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf800cdb0 con 0x7fcc08108650 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc081a0b30 con 0x7fcc08108650 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf8005020 con 0x7fcc08108650 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80116d0 con 0x7fcc08108650 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.365+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc081a1030 con 0x7fcc08108650 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.366+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc080fe8f0 con 0x7fcc08108650 2024-04-26T23:04:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.368+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fcbf80051d0 con 0x7fcc08108650 2024-04-26T23:04:44.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.368+0000 7fcbfffff640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcbf40a3c40 0x7fcbf40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.368+0000 7fcc06d9d640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcbf40a3c40 0x7fcbf40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.368+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fcbf80d0b70 con 0x7fcc08108650 2024-04-26T23:04:44.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.369+0000 7fcc06d9d640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcbf40a3c40 0x7fcbf40a6100 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcc0819a100 tx=0x7fcc00005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.372+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcbf8016030 con 0x7fcc08108650 2024-04-26T23:04:44.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:44 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/3551580454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:04:44.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:44 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/263617070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:04:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.558+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcc0802dd30 con 0x7fcc08108650 2024-04-26T23:04:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.559+0000 7fcbfffff640 1 -- 172.21.15.1:0/2459151608 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fcbf809d420 con 0x7fcc08108650 2024-04-26T23:04:44.559 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T23:04:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcbf40a3c40 msgr2=0x7fcbf40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcbf40a3c40 0x7fcbf40a6100 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcc0819a100 tx=0x7fcc00005e70 comp rx=0 tx=0).stop 2024-04-26T23:04:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 msgr2=0x7fcc081a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 0x7fcc081a01a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800eaa0 tx=0x7fcbf800ef60 comp rx=0 tx=0).stop 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 shutdown_connections 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fcbf40a3c40 0x7fcbf40a6100 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc08105d20 0x7fcc08199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc08108650 0x7fcc081a01a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 --2- 172.21.15.1:0/2459151608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc081033f0 0x7fcc08198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 >> 172.21.15.1:0/2459151608 conn(0x7fcc080fce00 msgr2=0x7fcc08105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 shutdown_connections 2024-04-26T23:04:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.561+0000 7fcc0d1a4640 1 -- 172.21.15.1:0/2459151608 wait complete. 2024-04-26T23:04:44.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:44 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/3551580454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:04:44.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:44 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/263617070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:04:44.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:04:44.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:44 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/3551580454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:04:44.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:44 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/263617070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:04:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 -- 172.21.15.1:0/1328645945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4101bd0 msgr2=0x7fbae4102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1328645945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4101bd0 0x7fbae4102030 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009900 tx=0x7fbacc02a480 comp rx=0 tx=0).stop 2024-04-26T23:04:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 -- 172.21.15.1:0/1328645945 shutdown_connections 2024-04-26T23:04:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1328645945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbae4102570 0x7fbae41029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1328645945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4101bd0 0x7fbae4102030 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1328645945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae4101270 0x7fbae4101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 -- 172.21.15.1:0/1328645945 >> 172.21.15.1:0/1328645945 conn(0x7fbae40fce10 msgr2=0x7fbae40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 -- 172.21.15.1:0/1328645945 shutdown_connections 2024-04-26T23:04:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.674+0000 7fbae96d9640 1 -- 172.21.15.1:0/1328645945 wait complete. 2024-04-26T23:04:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 Processor -- start 2024-04-26T23:04:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 -- start start 2024-04-26T23:04:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae4101270 0x7fbae419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbae4101bd0 0x7fbae419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 0x7fbae41a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbae41106b0 con 0x7fbae4102570 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbae4110530 con 0x7fbae4101bd0 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae96d9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbae4110830 con 0x7fbae4101270 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 0x7fbae41a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 0x7fbae41a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52908/0 (socket says 172.21.15.1:52908) 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 -- 172.21.15.1:0/1142418811 learned_addr learned my addr 172.21.15.1:0/1142418811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbadbfff640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbae4101bd0 0x7fbae419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae2ffd640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae4101270 0x7fbae419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 -- 172.21.15.1:0/1142418811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae4101270 msgr2=0x7fbae419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae4101270 0x7fbae419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 -- 172.21.15.1:0/1142418811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbae4101bd0 msgr2=0x7fbae419b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbae4101bd0 0x7fbae419b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.675+0000 7fbae37fe640 1 -- 172.21.15.1:0/1142418811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae41a2a60 con 0x7fbae4102570 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.676+0000 7fbae37fe640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 0x7fbae41a23f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00eaa0 tx=0x7fbadc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.676+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc00cdb0 con 0x7fbae4102570 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.676+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbadc005020 con 0x7fbae4102570 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.676+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0116d0 con 0x7fbae4102570 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.676+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae41a2d50 con 0x7fbae4102570 2024-04-26T23:04:44.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.676+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae41a31c0 con 0x7fbae4102570 2024-04-26T23:04:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.677+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae410bfc0 con 0x7fbae4102570 2024-04-26T23:04:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.678+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbadc0051d0 con 0x7fbae4102570 2024-04-26T23:04:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.679+0000 7fbae0ff9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbad00a3c40 0x7fbad00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.679+0000 7fbae2ffd640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbad00a3c40 0x7fbad00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.679+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fbadc0d0a50 con 0x7fbae4102570 2024-04-26T23:04:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.680+0000 7fbae2ffd640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbad00a3c40 0x7fbad00a6100 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009bd0 tx=0x7fbad4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.683+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbadc016030 con 0x7fbae4102570 2024-04-26T23:04:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.882+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbae402dad0 con 0x7fbae4102570 2024-04-26T23:04:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.883+0000 7fbae0ff9640 1 -- 172.21.15.1:0/1142418811 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+579 (secure 0 0 0) 0x7fbadc09cc90 con 0x7fbae4102570 2024-04-26T23:04:44.883 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2466076666 2024-04-27T22:40:29.666140+0000 2024-04-26T23:04:44.883 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3267707627 2024-04-27T22:40:29.666140+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3267707627 2024-04-27T22:40:29.666140+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2458055359 2024-04-27T22:40:29.666140+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/717752603 2024-04-27T22:40:02.607868+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4224349841 2024-04-27T22:40:02.607868+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/722669381 2024-04-27T22:40:02.607868+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3044795811 2024-04-27T22:40:02.607868+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/722669381 2024-04-27T22:40:02.607868+0000 2024-04-26T23:04:44.884 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3240847897 2024-04-27T22:40:29.666140+0000 2024-04-26T23:04:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-04-26T23:04:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbad00a3c40 msgr2=0x7fbad00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbad00a3c40 0x7fbad00a6100 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009bd0 tx=0x7fbad4005f90 comp rx=0 tx=0).stop 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 msgr2=0x7fbae41a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 0x7fbae41a23f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00eaa0 tx=0x7fbadc00ef60 comp rx=0 tx=0).stop 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 shutdown_connections 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fbad00a3c40 0x7fbad00a6100 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae4102570 0x7fbae41a23f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbae4101bd0 0x7fbae419b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 --2- 172.21.15.1:0/1142418811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae4101270 0x7fbae419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 >> 172.21.15.1:0/1142418811 conn(0x7fbae40fce10 msgr2=0x7fbae40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 shutdown_connections 2024-04-26T23:04:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:44.885+0000 7fbae96d9640 1 -- 172.21.15.1:0/1142418811 wait complete. 2024-04-26T23:04:44.896 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:44.896 DEBUG:teuthology.orchestra.run.smithi001:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:04:44.930 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T23:04:44.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:44.930 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:04:44.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:44.995 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T23:04:45.050 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:04:45.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:45.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:04:45.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:45.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-04-26T23:04:45.206 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T23:04:45.215 INFO:tasks.cephadm:Teardown begin 2024-04-26T23:04:45.215 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:04:45.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 -- 172.21.15.1:0/2035189760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102520 msgr2=0x7f5b74109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 --2- 172.21.15.1:0/2035189760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102520 0x7f5b74109dc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009ca0 tx=0x7f5b6c02a8c0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 -- 172.21.15.1:0/2035189760 shutdown_connections 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 --2- 172.21.15.1:0/2035189760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b74101b60 0x7f5b74101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 --2- 172.21.15.1:0/2035189760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102520 0x7f5b74109dc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 --2- 172.21.15.1:0/2035189760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74102f10 0x7f5b74103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 -- 172.21.15.1:0/2035189760 >> 172.21.15.1:0/2035189760 conn(0x7f5b740fc5f0 msgr2=0x7f5b740fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 -- 172.21.15.1:0/2035189760 shutdown_connections 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.320+0000 7f5b79e45640 1 -- 172.21.15.1:0/2035189760 wait complete. 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 Processor -- start 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 -- start start 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74101b60 0x7f5b74198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b74102520 0x7f5b74199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 0x7f5b741a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7410e4f0 con 0x7f5b74102f10 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7410e370 con 0x7f5b74101b60 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b79e45640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7410e670 con 0x7f5b74102520 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 0x7f5b741a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 0x7f5b741a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52920/0 (socket says 172.21.15.1:52920) 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.321+0000 7f5b73fff640 1 -- 172.21.15.1:0/45470409 learned_addr learned my addr 172.21.15.1:0/45470409 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b737fe640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74101b60 0x7f5b74198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b72ffd640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b74102520 0x7f5b74199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b73fff640 1 -- 172.21.15.1:0/45470409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b74102520 msgr2=0x7f5b74199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b73fff640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b74102520 0x7f5b74199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b73fff640 1 -- 172.21.15.1:0/45470409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74101b60 msgr2=0x7f5b74198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b73fff640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74101b60 0x7f5b74198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b73fff640 1 -- 172.21.15.1:0/45470409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b741a06f0 con 0x7f5b74102f10 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b737fe640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74101b60 0x7f5b74198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b73fff640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 0x7f5b741a0080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0097b0 tx=0x7f5b6c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c004240 con 0x7f5b74102f10 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b6c003e10 con 0x7f5b74102f10 2024-04-26T23:04:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c0305c0 con 0x7f5b74102f10 2024-04-26T23:04:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b741a0980 con 0x7f5b74102f10 2024-04-26T23:04:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.322+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b741a0ed0 con 0x7f5b74102f10 2024-04-26T23:04:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.323+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b74108ca0 con 0x7f5b74102f10 2024-04-26T23:04:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.324+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5b6c02f620 con 0x7f5b74102f10 2024-04-26T23:04:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.325+0000 7f5b70ff9640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5b5c0a3c40 0x7f5b5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.325+0000 7f5b737fe640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5b5c0a3c40 0x7f5b5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.325+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f5b6c0f1fb0 con 0x7f5b74102f10 2024-04-26T23:04:45.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.325+0000 7f5b737fe640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5b5c0a3c40 0x7f5b5c0a6100 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b7406aea0 tx=0x7f5b64005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.328+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5b6c0393b0 con 0x7f5b74102f10 2024-04-26T23:04:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:45 smithi186 ceph-mon[32774]: pgmap v804: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:45 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1438589913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:04:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:45 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2459151608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:04:45.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:45 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1142418811' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:04:45.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.553+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b7402dc90 con 0x7f5b74102f10 2024-04-26T23:04:45.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.553+0000 7f5b70ff9640 1 -- 172.21.15.1:0/45470409 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f5b6c0be910 con 0x7f5b74102f10 2024-04-26T23:04:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T23:04:45.554 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:49:42.273820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[14631,24473,24487,24475,24514]},"id":1}]} 2024-04-26T23:04:45.554 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-04-26T23:04:45.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5b5c0a3c40 msgr2=0x7f5b5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5b5c0a3c40 0x7f5b5c0a6100 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b7406aea0 tx=0x7f5b64005db0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 msgr2=0x7f5b741a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 0x7f5b741a0080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0097b0 tx=0x7f5b6c003c20 comp rx=0 tx=0).stop 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 shutdown_connections 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f5b5c0a3c40 0x7f5b5c0a6100 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b74102520 0x7f5b74199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b74102f10 0x7f5b741a0080 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 --2- 172.21.15.1:0/45470409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74101b60 0x7f5b74198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 >> 172.21.15.1:0/45470409 conn(0x7f5b740fc5f0 msgr2=0x7f5b74105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 shutdown_connections 2024-04-26T23:04:45.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.555+0000 7f5b79e45640 1 -- 172.21.15.1:0/45470409 wait complete. 2024-04-26T23:04:45.565 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:45 smithi001 ceph-mon[29435]: pgmap v804: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:45.565 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:45 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1438589913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:04:45.565 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:45 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2459151608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:04:45.566 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:45 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1142418811' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:04:45.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:45 smithi038 ceph-mon[32433]: pgmap v804: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T23:04:45.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:45 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1438589913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:04:45.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:45 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2459151608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:04:45.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:45 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1142418811' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:04:45.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- 172.21.15.1:0/4034331946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048101890 msgr2=0x7f1048101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- 172.21.15.1:0/4034331946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048101890 0x7f1048101cb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1040009900 tx=0x7f104002a410 comp rx=0 tx=0).stop 2024-04-26T23:04:45.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- 172.21.15.1:0/4034331946 shutdown_connections 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- 172.21.15.1:0/4034331946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1048103440 0x7f1048109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- 172.21.15.1:0/4034331946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1048102a80 0x7f1048102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- 172.21.15.1:0/4034331946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048101890 0x7f1048101cb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- 172.21.15.1:0/4034331946 >> 172.21.15.1:0/4034331946 conn(0x7f10480fce40 msgr2=0x7f10480ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- 172.21.15.1:0/4034331946 shutdown_connections 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- 172.21.15.1:0/4034331946 wait complete. 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 Processor -- start 2024-04-26T23:04:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- start start 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1048101890 0x7f104810c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1048102a80 0x7f104810a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 0x7f104810ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f104810e4e0 con 0x7f1048103440 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f104810e360 con 0x7f1048101890 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.669+0000 7f1050324640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f104810e660 con 0x7f1048102a80 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 0x7f104810ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 0x7f104810ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52938/0 (socket says 172.21.15.1:52938) 2024-04-26T23:04:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 -- 172.21.15.1:0/1882535939 learned_addr learned my addr 172.21.15.1:0/1882535939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104d8bf640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1048102a80 0x7f104810a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 -- 172.21.15.1:0/1882535939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1048102a80 msgr2=0x7f104810a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1048102a80 0x7f104810a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 -- 172.21.15.1:0/1882535939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1048101890 msgr2=0x7f104810c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e0c0640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1048101890 0x7f104810c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1048101890 0x7f104810c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 -- 172.21.15.1:0/1882535939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104810b620 con 0x7f1048103440 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f104e8c1640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 0x7f104810ad90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f103800eaa0 tx=0x7f103800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103800cdb0 con 0x7f1048103440 2024-04-26T23:04:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1038005020 con 0x7f1048103440 2024-04-26T23:04:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10380116d0 con 0x7f1048103440 2024-04-26T23:04:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481a2d30 con 0x7f1048103440 2024-04-26T23:04:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.670+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10481a31d0 con 0x7f1048103440 2024-04-26T23:04:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.671+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1048108c50 con 0x7f1048103440 2024-04-26T23:04:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.672+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f10380051d0 con 0x7f1048103440 2024-04-26T23:04:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.673+0000 7f10477fe640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10300a3c40 0x7f10300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.673+0000 7f104e0c0640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10300a3c40 0x7f10300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.673+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f10380d0850 con 0x7f1048103440 2024-04-26T23:04:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.673+0000 7f104e0c0640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10300a3c40 0x7f10300a6100 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f104002a930 tx=0x7f1040000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.677+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1038016030 con 0x7f1048103440 2024-04-26T23:04:45.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.907+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f104802dcf0 con 0x7f1048103440 2024-04-26T23:04:45.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.908+0000 7f10477fe640 1 -- 172.21.15.1:0/1882535939 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f103809d0a0 con 0x7f1048103440 2024-04-26T23:04:45.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T23:04:45.909 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2653371918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2653371918},{"type":"v1","addr":"172.21.15.1:6835","nonce":2653371918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1791309239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1791309239},{"type":"v1","addr":"172.21.15.1:6837","nonce":1791309239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T22:47:41.130042+0000","modified":"2024-04-26T22:49:42.273820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24514,"mds_1":24487,"mds_2":24475,"mds_3":24473,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6839/1512535471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":1512535471},{"type":"v1","addr":"172.21.15.1:6839","nonce":1512535471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6841/1622627716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":1622627716},{"type":"v1","addr":"172.21.15.1:6841","nonce":1622627716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14643":{"gid":14643,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/2490292790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2490292790},{"type":"v1","addr":"172.21.15.38:6839","nonce":2490292790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6833/3943944853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3943944853},{"type":"v1","addr":"172.21.15.38:6833","nonce":3943944853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/4165972801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4165972801},{"type":"v1","addr":"172.21.15.186:6833","nonce":4165972801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/1362256204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":1362256204},{"type":"v1","addr":"172.21.15.38:6835","nonce":1362256204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/1812912662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1812912662},{"type":"v1","addr":"172.21.15.38:6837","nonce":1812912662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6835/1379199020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1379199020},{"type":"v1","addr":"172.21.15.186:6835","nonce":1379199020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":1,"incarnation":17,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6837/786403012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":786403012},{"type":"v1","addr":"172.21.15.186:6837","nonce":786403012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.186:6839/3709434074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3709434074},{"type":"v1","addr":"172.21.15.186:6839","nonce":3709434074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24514,"qdb_cluster":[14631,24473,24487,24475,24514]},"id":1}]} 2024-04-26T23:04:45.911 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-04-26T23:04:45.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10300a3c40 msgr2=0x7f10300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10300a3c40 0x7f10300a6100 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f104002a930 tx=0x7f1040000bc0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 msgr2=0x7f104810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 0x7f104810ad90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f103800eaa0 tx=0x7f103800ef60 comp rx=0 tx=0).stop 2024-04-26T23:04:45.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 shutdown_connections 2024-04-26T23:04:45.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f10300a3c40 0x7f10300a6100 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1048102a80 0x7f104810a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048103440 0x7f104810ad90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 --2- 172.21.15.1:0/1882535939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1048101890 0x7f104810c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 >> 172.21.15.1:0/1882535939 conn(0x7f10480fce40 msgr2=0x7f1048105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 shutdown_connections 2024-04-26T23:04:45.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:45.910+0000 7f1050324640 1 -- 172.21.15.1:0/1882535939 wait complete. 2024-04-26T23:04:45.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:46.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 -- 172.21.15.1:0/2994954505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024102a80 msgr2=0x7fa024102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/2994954505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024102a80 0x7fa024102f00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa018009900 tx=0x7fa01802a410 comp rx=0 tx=0).stop 2024-04-26T23:04:46.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 -- 172.21.15.1:0/2994954505 shutdown_connections 2024-04-26T23:04:46.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/2994954505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024103440 0x7fa024109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/2994954505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024102a80 0x7fa024102f00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/2994954505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 0x7fa024101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 -- 172.21.15.1:0/2994954505 >> 172.21.15.1:0/2994954505 conn(0x7fa0240fce00 msgr2=0x7fa0240ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 -- 172.21.15.1:0/2994954505 shutdown_connections 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 -- 172.21.15.1:0/2994954505 wait complete. 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 Processor -- start 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 -- start start 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 0x7fa024198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024102a80 0x7fa0241991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.024+0000 7fa02aa1d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 0x7fa0241a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa02aa1d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa02410d3a0 con 0x7fa024103440 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa02aa1d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa02410d220 con 0x7fa024101890 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa02aa1d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa02410d520 con 0x7fa024102a80 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 0x7fa0241a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 0x7fa0241a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52952/0 (socket says 172.21.15.1:52952) 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 -- 172.21.15.1:0/903562334 learned_addr learned my addr 172.21.15.1:0/903562334 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa0237fe640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024102a80 0x7fa0241991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa023fff640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 0x7fa024198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 -- 172.21.15.1:0/903562334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024102a80 msgr2=0x7fa0241991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024102a80 0x7fa0241991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 -- 172.21.15.1:0/903562334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 msgr2=0x7fa024198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 0x7fa024198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 -- 172.21.15.1:0/903562334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0241a08c0 con 0x7fa024103440 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa023fff640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 0x7fa024198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa028fba640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 0x7fa0241a0220 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009fd0 tx=0x7fa01c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c017070 con 0x7fa024103440 2024-04-26T23:04:46.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0241a0bb0 con 0x7fa024103440 2024-04-26T23:04:46.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa01c003e20 con 0x7fa024103440 2024-04-26T23:04:46.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c010460 con 0x7fa024103440 2024-04-26T23:04:46.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.025+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a1080 con 0x7fa024103440 2024-04-26T23:04:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.026+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0241a1250 con 0x7fa024103440 2024-04-26T23:04:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.028+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa01c003970 con 0x7fa024103440 2024-04-26T23:04:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.028+0000 7fa0217fa640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa00c0a3c40 0x7fa00c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.028+0000 7fa023fff640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa00c0a3c40 0x7fa00c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.028+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fa01c0d0370 con 0x7fa024103440 2024-04-26T23:04:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.029+0000 7fa023fff640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa00c0a3c40 0x7fa00c0a6100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa024199b50 tx=0x7fa014005e90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:46.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.032+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa01c09cc20 con 0x7fa024103440 2024-04-26T23:04:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.223+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0241a15e0 con 0x7fa024103440 2024-04-26T23:04:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.224+0000 7fa0217fa640 1 -- 172.21.15.1:0/903562334 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22426 (secure 0 0 0) 0x7fa01c09c520 con 0x7fa024103440 2024-04-26T23:04:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T23:04:46.225 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":87,"fsid":"bb8b501a-041d-11ef-bc93-c7b262605968","created":"2024-04-26T22:39:26.836716+0000","modified":"2024-04-26T22:48:30.916101+0000","last_up_change":"2024-04-26T22:46:47.796539+0000","last_in_change":"2024-04-26T22:46:27.156372+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T22:42:55.759310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T22:47:39.631594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T22:47:40.081969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5213229d-410b-4bd5-9fb0-fb58a13e3322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6803","nonce":1864154398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6805","nonce":1864154398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6809","nonce":1864154398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1864154398},{"type":"v1","addr":"172.21.15.1:6807","nonce":1864154398}]},"public_addr":"172.21.15.1:6803/1864154398","cluster_addr":"172.21.15.1:6805/1864154398","heartbeat_back_addr":"172.21.15.1:6809/1864154398","heartbeat_front_addr":"172.21.15.1:6807/1864154398","state":["exists","up"]},{"osd":1,"uuid":"efa8c529-7d81-4be2-90ee-2ce0cc461c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6801","nonce":156758275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6803","nonce":156758275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6807","nonce":156758275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":156758275},{"type":"v1","addr":"172.21.15.38:6805","nonce":156758275}]},"public_addr":"172.21.15.38:6801/156758275","cluster_addr":"172.21.15.38:6803/156758275","heartbeat_back_addr":"172.21.15.38:6807/156758275","heartbeat_front_addr":"172.21.15.38:6805/156758275","state":["exists","up"]},{"osd":2,"uuid":"7d043997-5257-4a78-99a6-4e3483360b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6801","nonce":2937417141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6803","nonce":2937417141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6807","nonce":2937417141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2937417141},{"type":"v1","addr":"172.21.15.186:6805","nonce":2937417141}]},"public_addr":"172.21.15.186:6801/2937417141","cluster_addr":"172.21.15.186:6803/2937417141","heartbeat_back_addr":"172.21.15.186:6807/2937417141","heartbeat_front_addr":"172.21.15.186:6805/2937417141","state":["exists","up"]},{"osd":3,"uuid":"ed1bef0e-ba55-4c4a-9423-0024c218b1dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6811","nonce":10840892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6813","nonce":10840892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6817","nonce":10840892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":10840892},{"type":"v1","addr":"172.21.15.1:6815","nonce":10840892}]},"public_addr":"172.21.15.1:6811/10840892","cluster_addr":"172.21.15.1:6813/10840892","heartbeat_back_addr":"172.21.15.1:6817/10840892","heartbeat_front_addr":"172.21.15.1:6815/10840892","state":["exists","up"]},{"osd":4,"uuid":"98421e8d-87a9-4a37-9415-d806e7d31651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6809","nonce":414739057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6811","nonce":414739057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6815","nonce":414739057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":414739057},{"type":"v1","addr":"172.21.15.38:6813","nonce":414739057}]},"public_addr":"172.21.15.38:6809/414739057","cluster_addr":"172.21.15.38:6811/414739057","heartbeat_back_addr":"172.21.15.38:6815/414739057","heartbeat_front_addr":"172.21.15.38:6813/414739057","state":["exists","up"]},{"osd":5,"uuid":"f6b59471-baf0-4b9b-8756-43142ee043fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6809","nonce":1470594715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6811","nonce":1470594715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6815","nonce":1470594715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1470594715},{"type":"v1","addr":"172.21.15.186:6813","nonce":1470594715}]},"public_addr":"172.21.15.186:6809/1470594715","cluster_addr":"172.21.15.186:6811/1470594715","heartbeat_back_addr":"172.21.15.186:6815/1470594715","heartbeat_front_addr":"172.21.15.186:6813/1470594715","state":["exists","up"]},{"osd":6,"uuid":"cc08c46f-8ce2-448c-a9fb-74b7559b0d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6819","nonce":244390713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6821","nonce":244390713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6825","nonce":244390713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":244390713},{"type":"v1","addr":"172.21.15.1:6823","nonce":244390713}]},"public_addr":"172.21.15.1:6819/244390713","cluster_addr":"172.21.15.1:6821/244390713","heartbeat_back_addr":"172.21.15.1:6825/244390713","heartbeat_front_addr":"172.21.15.1:6823/244390713","state":["exists","up"]},{"osd":7,"uuid":"579e3ab3-4803-4c10-835f-266ce4f811ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6817","nonce":519750991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6819","nonce":519750991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6823","nonce":519750991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":519750991},{"type":"v1","addr":"172.21.15.38:6821","nonce":519750991}]},"public_addr":"172.21.15.38:6817/519750991","cluster_addr":"172.21.15.38:6819/519750991","heartbeat_back_addr":"172.21.15.38:6823/519750991","heartbeat_front_addr":"172.21.15.38:6821/519750991","state":["exists","up"]},{"osd":8,"uuid":"0e754df8-e30f-40bf-b754-351c414e8767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6817","nonce":955115503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6819","nonce":955115503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6823","nonce":955115503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":955115503},{"type":"v1","addr":"172.21.15.186:6821","nonce":955115503}]},"public_addr":"172.21.15.186:6817/955115503","cluster_addr":"172.21.15.186:6819/955115503","heartbeat_back_addr":"172.21.15.186:6823/955115503","heartbeat_front_addr":"172.21.15.186:6821/955115503","state":["exists","up"]},{"osd":9,"uuid":"f781b994-298e-413f-9391-b9de190549f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6827","nonce":2190648434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6829","nonce":2190648434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6833","nonce":2190648434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":2190648434},{"type":"v1","addr":"172.21.15.1:6831","nonce":2190648434}]},"public_addr":"172.21.15.1:6827/2190648434","cluster_addr":"172.21.15.1:6829/2190648434","heartbeat_back_addr":"172.21.15.1:6833/2190648434","heartbeat_front_addr":"172.21.15.1:6831/2190648434","state":["exists","up"]},{"osd":10,"uuid":"9cbc770f-5853-41c8-9be5-c5b2dcec7fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6825","nonce":1475724574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6827","nonce":1475724574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6831","nonce":1475724574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1475724574},{"type":"v1","addr":"172.21.15.38:6829","nonce":1475724574}]},"public_addr":"172.21.15.38:6825/1475724574","cluster_addr":"172.21.15.38:6827/1475724574","heartbeat_back_addr":"172.21.15.38:6831/1475724574","heartbeat_front_addr":"172.21.15.38:6829/1475724574","state":["exists","up"]},{"osd":11,"uuid":"7e7a126e-d734-4b97-aa0e-6a547cb69575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6825","nonce":63751610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6827","nonce":63751610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6831","nonce":63751610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":63751610},{"type":"v1","addr":"172.21.15.186:6829","nonce":63751610}]},"public_addr":"172.21.15.186:6825/63751610","cluster_addr":"172.21.15.186:6827/63751610","heartbeat_back_addr":"172.21.15.186:6831/63751610","heartbeat_front_addr":"172.21.15.186:6829/63751610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:03.711412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:28.343591+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:42:52.158847+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:17.162562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:43:42.592663+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:07.461307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:33.631199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:44:59.887197+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:25.265410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:45:52.496795+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:19.181394+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T22:46:45.775229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2466076666":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6801/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:6800/3267707627":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/2458055359":"2024-04-27T22:40:29.666140+0000","172.21.15.1:0/717752603":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/4224349841":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6800/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3044795811":"2024-04-27T22:40:02.607868+0000","172.21.15.1:6801/722669381":"2024-04-27T22:40:02.607868+0000","172.21.15.1:0/3240847897":"2024-04-27T22:40:29.666140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:46.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa00c0a3c40 msgr2=0x7fa00c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa00c0a3c40 0x7fa00c0a6100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa024199b50 tx=0x7fa014005e90 comp rx=0 tx=0).stop 2024-04-26T23:04:46.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 msgr2=0x7fa0241a0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 0x7fa0241a0220 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009fd0 tx=0x7fa01c00b260 comp rx=0 tx=0).stop 2024-04-26T23:04:46.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 shutdown_connections 2024-04-26T23:04:46.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fa00c0a3c40 0x7fa00c0a6100 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024102a80 0x7fa0241991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa024103440 0x7fa0241a0220 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 --2- 172.21.15.1:0/903562334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa024101890 0x7fa024198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 >> 172.21.15.1:0/903562334 conn(0x7fa0240fce00 msgr2=0x7fa024069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:46.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 shutdown_connections 2024-04-26T23:04:46.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.226+0000 7fa02aa1d640 1 -- 172.21.15.1:0/903562334 wait complete. 2024-04-26T23:04:46.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:04:46.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 -- 172.21.15.1:0/3819486567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0102a60 msgr2=0x7f16d0102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 --2- 172.21.15.1:0/3819486567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0102a60 0x7f16d0102ee0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009900 tx=0x7f16b802a410 comp rx=0 tx=0).stop 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 -- 172.21.15.1:0/3819486567 shutdown_connections 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 --2- 172.21.15.1:0/3819486567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 0x7f16d0109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 --2- 172.21.15.1:0/3819486567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0102a60 0x7f16d0102ee0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 --2- 172.21.15.1:0/3819486567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d0101870 0x7f16d0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 -- 172.21.15.1:0/3819486567 >> 172.21.15.1:0/3819486567 conn(0x7f16d00fce00 msgr2=0x7f16d00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:46.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 -- 172.21.15.1:0/3819486567 shutdown_connections 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.341+0000 7f16d5911640 1 -- 172.21.15.1:0/3819486567 wait complete. 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 Processor -- start 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 -- start start 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 0x7f16d0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d0102a60 0x7f16d0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 0x7f16d01a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16d010d3a0 con 0x7f16d0101870 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16d010d220 con 0x7f16d0103420 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16d5911640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16d010d520 con 0x7f16d0102a60 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 0x7f16d0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 0x7f16d0198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52966/0 (socket says 172.21.15.1:52966) 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 -- 172.21.15.1:0/1596775508 learned_addr learned my addr 172.21.15.1:0/1596775508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16cf7fe640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 0x7f16d01a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16c7fff640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d0102a60 0x7f16d0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 -- 172.21.15.1:0/1596775508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d0102a60 msgr2=0x7f16d0199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d0102a60 0x7f16d0199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 -- 172.21.15.1:0/1596775508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 msgr2=0x7f16d01a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 0x7f16d01a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16ceffd640 1 -- 172.21.15.1:0/1596775508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d01a07f0 con 0x7f16d0101870 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.342+0000 7f16cf7fe640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 0x7f16d01a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:46.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.343+0000 7f16ceffd640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 0x7f16d0198bd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c970 tx=0x7f16c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.343+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0004220 con 0x7f16d0101870 2024-04-26T23:04:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.343+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16c00043c0 con 0x7f16d0101870 2024-04-26T23:04:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.343+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d01a0ae0 con 0x7f16d0101870 2024-04-26T23:04:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.343+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0004a00 con 0x7f16d0101870 2024-04-26T23:04:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.343+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d01a0fe0 con 0x7f16d0101870 2024-04-26T23:04:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.344+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d01a1210 con 0x7f16d0101870 2024-04-26T23:04:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.345+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f16c00079b0 con 0x7f16d0101870 2024-04-26T23:04:46.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.346+0000 7f16d490f640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f16bc0a3c40 0x7f16bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.346+0000 7f16c7fff640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f16bc0a3c40 0x7f16bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.346+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f16c00d07e0 con 0x7f16d0101870 2024-04-26T23:04:46.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.346+0000 7f16c7fff640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f16bc0a3c40 0x7f16bc0a6100 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f16d019a0e0 tx=0x7f16b80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:46.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.349+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f16c0014030 con 0x7f16d0101870 2024-04-26T23:04:46.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/45470409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:04:46.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1882535939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:04:46.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.574+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f16d01a13e0 con 0x7f16d0101870 2024-04-26T23:04:46.576 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/45470409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:04:46.576 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1882535939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:04:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.576+0000 7f16d490f640 1 -- 172.21.15.1:0/1596775508 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7f16c009d030 con 0x7f16d0101870 2024-04-26T23:04:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:04:46.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f16bc0a3c40 msgr2=0x7f16bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f16bc0a3c40 0x7f16bc0a6100 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f16d019a0e0 tx=0x7f16b80314a0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 msgr2=0x7f16d0198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 0x7f16d0198bd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c970 tx=0x7f16c000ce30 comp rx=0 tx=0).stop 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 shutdown_connections 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f16bc0a3c40 0x7f16bc0a6100 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d0102a60 0x7f16d0199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d0103420 0x7f16d01a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 --2- 172.21.15.1:0/1596775508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d0101870 0x7f16d0198bd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 >> 172.21.15.1:0/1596775508 conn(0x7f16d00fce00 msgr2=0x7f16d006b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 shutdown_connections 2024-04-26T23:04:46.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.580+0000 7f16d5911640 1 -- 172.21.15.1:0/1596775508 wait complete. 2024-04-26T23:04:46.595 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:04:46.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/45470409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:04:46.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1882535939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:04:46.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 -- 172.21.15.1:0/187310185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530101870 msgr2=0x7f7530101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 --2- 172.21.15.1:0/187310185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530101870 0x7f7530101c90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f752c009900 tx=0x7f752c02a410 comp rx=0 tx=0).stop 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 -- 172.21.15.1:0/187310185 shutdown_connections 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 --2- 172.21.15.1:0/187310185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7530102a60 0x7f7530102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 --2- 172.21.15.1:0/187310185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530103420 0x7f7530109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 --2- 172.21.15.1:0/187310185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530101870 0x7f7530101c90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 -- 172.21.15.1:0/187310185 >> 172.21.15.1:0/187310185 conn(0x7f75300fce00 msgr2=0x7f75300ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 -- 172.21.15.1:0/187310185 shutdown_connections 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 -- 172.21.15.1:0/187310185 wait complete. 2024-04-26T23:04:46.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 Processor -- start 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.705+0000 7f7537820640 1 -- start start 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7537820640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7530101870 0x7f753010c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7537820640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530102a60 0x7f753010a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7537820640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 0x7f753010ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7537820640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f753010e4c0 con 0x7f7530103420 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7537820640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f753010e340 con 0x7f7530102a60 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7537820640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f753010e640 con 0x7f7530101870 2024-04-26T23:04:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 0x7f753010ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 0x7f753010ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53030/0 (socket says 172.21.15.1:53030) 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 -- 172.21.15.1:0/2429411874 learned_addr learned my addr 172.21.15.1:0/2429411874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7534dbb640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530102a60 0x7f753010a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f75355bc640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7530101870 0x7f753010c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 -- 172.21.15.1:0/2429411874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7530101870 msgr2=0x7f753010c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7530101870 0x7f753010c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 -- 172.21.15.1:0/2429411874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530102a60 msgr2=0x7f753010a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530102a60 0x7f753010a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 -- 172.21.15.1:0/2429411874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753010b600 con 0x7f7530103420 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7534dbb640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530102a60 0x7f753010a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f7535dbd640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 0x7f753010ad70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7528009fd0 tx=0x7f752800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.706+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528017070 con 0x7f7530103420 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.707+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7528003e20 con 0x7f7530103420 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.707+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75301a2d90 con 0x7f7530103420 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.707+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75280106c0 con 0x7f7530103420 2024-04-26T23:04:46.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.707+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75301a31d0 con 0x7f7530103420 2024-04-26T23:04:46.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.707+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7530108ca0 con 0x7f7530103420 2024-04-26T23:04:46.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.709+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f7528004140 con 0x7f7530103420 2024-04-26T23:04:46.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.710+0000 7f75267fc640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f751c0a3c40 0x7f751c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:46.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.710+0000 7f75355bc640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f751c0a3c40 0x7f751c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:46.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.710+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8612+0+0 (secure 0 0 0) 0x7f75280d0420 con 0x7f7530103420 2024-04-26T23:04:46.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.710+0000 7f75355bc640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f751c0a3c40 0x7f751c0a6100 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f752c000c00 tx=0x7f752c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:46.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.714+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f75280047e0 con 0x7f7530103420 2024-04-26T23:04:46.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a[29431]: 2024-04-26T23:04:46.575+0000 7f942318a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:04:46.846 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a[69459]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:46.846 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a[69459]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:46.847 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-mds[69463]: -- 172.21.15.1:0/3431221333 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a185283a40 con 0x55a185256c00 2024-04-26T23:04:46.847 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a[69459]: starting mds.a at 2024-04-26T23:04:46.847 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g[69114]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:46.847 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g[69114]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:46.847 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-mds[69118]: -- 172.21.15.1:0/1257574278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e0e7459a40 con 0x55e0e742cc00 2024-04-26T23:04:46.847 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:04:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g[69114]: starting mds.g at 2024-04-26T23:04:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:46.940+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f753002db70 con 0x7f7530103420 2024-04-26T23:04:46.985 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:46.985 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:46.985 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-mds[63178]: -- 172.21.15.186:0/768130926 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a1947bba40 con 0x55a19478ec00 2024-04-26T23:04:46.986 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: starting mds.i at 2024-04-26T23:04:46.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c[62862]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:46.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c[62862]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:46.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-mds[62866]: -- 172.21.15.186:0/530183279 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a769f5fa40 con 0x55a769f32c00 2024-04-26T23:04:46.986 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c[62862]: starting mds.c at 2024-04-26T23:04:46.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l[62252]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:46.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l[62252]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:46.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-mds[62256]: -- 172.21.15.186:0/7073698 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af0fa69a40 con 0x55af0fa3c800 2024-04-26T23:04:46.987 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l[62252]: starting mds.l at 2024-04-26T23:04:46.987 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f[62556]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:46.988 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f[62556]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:46.988 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-mds[62560]: -- 172.21.15.186:0/1693150476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558be3e57a40 con 0x558be3e2ac00 2024-04-26T23:04:46.988 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:04:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f[62556]: starting mds.f at 2024-04-26T23:04:47.101 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k[61452]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:47.101 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k[61452]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:47.102 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mds[61456]: -- 172.21.15.38:0/3007475848 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b9096eda40 con 0x55b9096c0c00 2024-04-26T23:04:47.102 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k[61452]: starting mds.k at 2024-04-26T23:04:47.102 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e[61773]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:47.102 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e[61773]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:47.102 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mds[61777]: -- 172.21.15.38:0/48555160 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0fa371a40 con 0x55d0fa344800 2024-04-26T23:04:47.103 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e[61773]: starting mds.e at 2024-04-26T23:04:47.103 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h[62082]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:47.103 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h[62082]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:47.103 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mds[62086]: -- 172.21.15.38:0/1299586732 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5623965ea4e0 con 0x5623965c0800 2024-04-26T23:04:47.103 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mds[62086]: -- 172.21.15.38:0/1299586732 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623965eda40 con 0x5623965c0800 2024-04-26T23:04:47.103 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h[62082]: starting mds.h at 2024-04-26T23:04:47.104 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b[62411]: ignoring --setuser ceph since I am not root 2024-04-26T23:04:47.104 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b[62411]: ignoring --setgroup ceph since I am not root 2024-04-26T23:04:47.104 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-mds[62415]: -- 172.21.15.38:0/1580326167 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638cf34da40 con 0x5638cf321000 2024-04-26T23:04:47.104 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:04:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b[62411]: starting mds.b at 2024-04-26T23:04:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: pgmap v805: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T23:04:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/903562334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:04:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1596775508' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T23:04:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T23:04:47.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:04:47.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/1596775508' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T23:04:47.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:04:47.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-26T23:04:47.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:47 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2429411874' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:04:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.578+0000 7f75267fc640 1 -- 172.21.15.1:0/2429411874 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f752809cb20 con 0x7f7530103420 2024-04-26T23:04:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f751c0a3c40 msgr2=0x7f751c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f751c0a3c40 0x7f751c0a6100 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f752c000c00 tx=0x7f752c0314a0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 msgr2=0x7f753010ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 0x7f753010ad70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7528009fd0 tx=0x7f752800b260 comp rx=0 tx=0).stop 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 shutdown_connections 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f751c0a3c40 0x7f751c0a6100 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7530103420 0x7f753010ad70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7530102a60 0x7f753010a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 --2- 172.21.15.1:0/2429411874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7530101870 0x7f753010c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 >> 172.21.15.1:0/2429411874 conn(0x7f75300fce00 msgr2=0x7f7530105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 shutdown_connections 2024-04-26T23:04:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.581+0000 7f7537820640 1 -- 172.21.15.1:0/2429411874 wait complete. 2024-04-26T23:04:47.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: pgmap v805: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T23:04:47.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/903562334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:04:47.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1596775508' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T23:04:47.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T23:04:47.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:04:47.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/1596775508' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T23:04:47.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:04:47.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-26T23:04:47.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:47 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2429411874' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:04:47.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:04:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: pgmap v805: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-26T23:04:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/903562334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:04:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1596775508' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T23:04:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T23:04:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:04:47.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/1596775508' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T23:04:47.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:04:47.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-26T23:04:47.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:47 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2429411874' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:04:47.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 -- 172.21.15.1:0/1335250493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 msgr2=0x7fd420105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 --2- 172.21.15.1:0/1335250493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd420105fc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd410009900 tx=0x7fd41002a410 comp rx=0 tx=0).stop 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 -- 172.21.15.1:0/1335250493 shutdown_connections 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 --2- 172.21.15.1:0/1335250493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd420106500 0x7fd420106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 --2- 172.21.15.1:0/1335250493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd420105fc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 --2- 172.21.15.1:0/1335250493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4201011e0 0x7fd420103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 -- 172.21.15.1:0/1335250493 >> 172.21.15.1:0/1335250493 conn(0x7fd4200fce00 msgr2=0x7fd4200ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 -- 172.21.15.1:0/1335250493 shutdown_connections 2024-04-26T23:04:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.711+0000 7fd4280b8640 1 -- 172.21.15.1:0/1335250493 wait complete. 2024-04-26T23:04:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 Processor -- start 2024-04-26T23:04:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 -- start start 2024-04-26T23:04:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4201011e0 0x7fd42019f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd42019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd420106500 0x7fd4201a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd420114ac0 con 0x7fd420103b50 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd420114940 con 0x7fd420106500 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd4280b8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd420114c40 con 0x7fd4201011e0 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd42019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd42019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53048/0 (socket says 172.21.15.1:53048) 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 -- 172.21.15.1:0/421584002 learned_addr learned my addr 172.21.15.1:0/421584002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425e54640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4201011e0 0x7fd42019f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd426655640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd420106500 0x7fd4201a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 -- 172.21.15.1:0/421584002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4201011e0 msgr2=0x7fd42019f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4201011e0 0x7fd42019f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 -- 172.21.15.1:0/421584002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd420106500 msgr2=0x7fd4201a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.712+0000 7fd425653640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd420106500 0x7fd4201a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd425653640 1 -- 172.21.15.1:0/421584002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4201a6e10 con 0x7fd420103b50 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd425653640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd42019f730 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd410005940 tx=0x7fd410031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41003f030 con 0x7fd420103b50 2024-04-26T23:04:47.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd410003aa0 con 0x7fd420103b50 2024-04-26T23:04:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4201a70a0 con 0x7fd420103b50 2024-04-26T23:04:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd410035070 con 0x7fd420103b50 2024-04-26T23:04:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.713+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4201a7590 con 0x7fd420103b50 2024-04-26T23:04:47.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.714+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4200fe8f0 con 0x7fd420103b50 2024-04-26T23:04:47.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.715+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd410042030 con 0x7fd420103b50 2024-04-26T23:04:47.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.716+0000 7fd416ffd640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd40c0a3c40 0x7fd40c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.716+0000 7fd425e54640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd40c0a3c40 0x7fd40c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.717+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8612+0+0 (secure 0 0 0) 0x7fd4100f13f0 con 0x7fd420103b50 2024-04-26T23:04:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.717+0000 7fd425e54640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd40c0a3c40 0x7fd40c0a6100 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009b60 tx=0x7fd41c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.721+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd4100bdaf0 con 0x7fd420103b50 2024-04-26T23:04:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:47.907+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd42002dad0 con 0x7fd420103b50 2024-04-26T23:04:48.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T23:04:48.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T23:04:48.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: Cluster is now healthy 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2429411874' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6838/2503198,v1:172.21.15.1:6839/2503198] up:boot 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6832/1403201513,v1:172.21.15.186:6834/1403201513] up:boot 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6833/1659445847,v1:172.21.15.186:6835/1659445847] up:boot 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.1:6840/439028064,v1:172.21.15.1:6841/439028064] up:boot 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6836/155894229,v1:172.21.15.186:6837/155894229] up:boot 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.186:6838/216455713,v1:172.21.15.186:6839/216455713] up:boot 2024-04-26T23:04:48.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6832/23197935,v1:172.21.15.38:6836/23197935] up:boot 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6834/3907600699,v1:172.21.15.38:6838/3907600699] up:boot 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6833/1059057775,v1:172.21.15.38:6837/1059057775] up:boot 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: mds.? [v2:172.21.15.38:6835/2429388963,v1:172.21.15.38:6839/2429388963] up:boot 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: fsmap 12 up:standby 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:04:48.487 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:04:48.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:04:48.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:04:48.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:04:48.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:04:48.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:04:48.488 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:48 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/421584002' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T23:04:48.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.589+0000 7fd416ffd640 1 -- 172.21.15.1:0/421584002 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) ==== 181+0+0 (secure 0 0 0) 0x7fd4100bd3f0 con 0x7fd420103b50 2024-04-26T23:04:48.589 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:04:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd40c0a3c40 msgr2=0x7fd40c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd40c0a3c40 0x7fd40c0a6100 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009b60 tx=0x7fd41c005f90 comp rx=0 tx=0).stop 2024-04-26T23:04:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 msgr2=0x7fd42019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd42019f730 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd410005940 tx=0x7fd410031a50 comp rx=0 tx=0).stop 2024-04-26T23:04:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 shutdown_connections 2024-04-26T23:04:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7fd40c0a3c40 0x7fd40c0a6100 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd420106500 0x7fd4201a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd420103b50 0x7fd42019f730 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 --2- 172.21.15.1:0/421584002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4201011e0 0x7fd42019f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 >> 172.21.15.1:0/421584002 conn(0x7fd4200fce00 msgr2=0x7fd420101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.593+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 shutdown_connections 2024-04-26T23:04:48.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.594+0000 7fd4280b8640 1 -- 172.21.15.1:0/421584002 wait complete. 2024-04-26T23:04:48.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T23:04:48.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T23:04:48.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: Cluster is now healthy 2024-04-26T23:04:48.596 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2429411874' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:04:48.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6838/2503198,v1:172.21.15.1:6839/2503198] up:boot 2024-04-26T23:04:48.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6832/1403201513,v1:172.21.15.186:6834/1403201513] up:boot 2024-04-26T23:04:48.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6833/1659445847,v1:172.21.15.186:6835/1659445847] up:boot 2024-04-26T23:04:48.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.1:6840/439028064,v1:172.21.15.1:6841/439028064] up:boot 2024-04-26T23:04:48.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6836/155894229,v1:172.21.15.186:6837/155894229] up:boot 2024-04-26T23:04:48.597 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.186:6838/216455713,v1:172.21.15.186:6839/216455713] up:boot 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6832/23197935,v1:172.21.15.38:6836/23197935] up:boot 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6834/3907600699,v1:172.21.15.38:6838/3907600699] up:boot 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6833/1059057775,v1:172.21.15.38:6837/1059057775] up:boot 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: mds.? [v2:172.21.15.38:6835/2429388963,v1:172.21.15.38:6839/2429388963] up:boot 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: fsmap 12 up:standby 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:04:48.598 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:04:48.599 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:48 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/421584002' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T23:04:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T23:04:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T23:04:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: Cluster is now healthy 2024-04-26T23:04:48.601 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2429411874' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6838/2503198,v1:172.21.15.1:6839/2503198] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6832/1403201513,v1:172.21.15.186:6834/1403201513] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6833/1659445847,v1:172.21.15.186:6835/1659445847] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.1:6840/439028064,v1:172.21.15.1:6841/439028064] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6836/155894229,v1:172.21.15.186:6837/155894229] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.186:6838/216455713,v1:172.21.15.186:6839/216455713] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6832/23197935,v1:172.21.15.38:6836/23197935] up:boot 2024-04-26T23:04:48.602 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6834/3907600699,v1:172.21.15.38:6838/3907600699] up:boot 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6833/1059057775,v1:172.21.15.38:6837/1059057775] up:boot 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: mds.? [v2:172.21.15.38:6835/2429388963,v1:172.21.15.38:6839/2429388963] up:boot 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: fsmap 12 up:standby 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:04:48.603 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:04:48.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:04:48.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:04:48.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:04:48.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='mgr.14152 172.21.15.1:0/4050829205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:04:48.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:48 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/421584002' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T23:04:48.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:04:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 -- 172.21.15.1:0/2843234823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0101850 msgr2=0x7f59c0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 --2- 172.21.15.1:0/2843234823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0101850 0x7f59c0101c70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009900 tx=0x7f59b002a410 comp rx=0 tx=0).stop 2024-04-26T23:04:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 -- 172.21.15.1:0/2843234823 shutdown_connections 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 --2- 172.21.15.1:0/2843234823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59c0103400 0x7f59c0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 --2- 172.21.15.1:0/2843234823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c0102a40 0x7f59c0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 --2- 172.21.15.1:0/2843234823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0101850 0x7f59c0101c70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 -- 172.21.15.1:0/2843234823 >> 172.21.15.1:0/2843234823 conn(0x7f59c00fcda0 msgr2=0x7f59c00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 -- 172.21.15.1:0/2843234823 shutdown_connections 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 -- 172.21.15.1:0/2843234823 wait complete. 2024-04-26T23:04:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.722+0000 7f59c6028640 1 Processor -- start 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 -- start start 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c0101850 0x7f59c0198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59c0102a40 0x7f59c01990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 0x7f59c01a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d350 con 0x7f59c0103400 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d1d0 con 0x7f59c0101850 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59c6028640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d4d0 con 0x7f59c0102a40 2024-04-26T23:04:48.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 0x7f59c01a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 0x7f59c01a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53074/0 (socket says 172.21.15.1:53074) 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 -- 172.21.15.1:0/2571385606 learned_addr learned my addr 172.21.15.1:0/2571385606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bf7fe640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c0101850 0x7f59c0198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59beffd640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59c0102a40 0x7f59c01990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 -- 172.21.15.1:0/2571385606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59c0102a40 msgr2=0x7f59c01990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59c0102a40 0x7f59c01990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 -- 172.21.15.1:0/2571385606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c0101850 msgr2=0x7f59c0198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c0101850 0x7f59c0198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.723+0000 7f59bffff640 1 -- 172.21.15.1:0/2571385606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c01a07a0 con 0x7f59c0103400 2024-04-26T23:04:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.724+0000 7f59bffff640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 0x7f59c01a0130 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f59b800eaa0 tx=0x7f59b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.724+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b800cdb0 con 0x7f59c0103400 2024-04-26T23:04:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.724+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b8005020 con 0x7f59c0103400 2024-04-26T23:04:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.724+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c01a0a90 con 0x7f59c0103400 2024-04-26T23:04:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.724+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b80116d0 con 0x7f59c0103400 2024-04-26T23:04:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.724+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c01a0f90 con 0x7f59c0103400 2024-04-26T23:04:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.725+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c01a1280 con 0x7f59c0103400 2024-04-26T23:04:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.726+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f59b80051d0 con 0x7f59c0103400 2024-04-26T23:04:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.726+0000 7f59bcff9640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f59a80a3c40 0x7f59a80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.727+0000 7f59bf7fe640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f59a80a3c40 0x7f59a80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.727+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8116+0+0 (secure 0 0 0) 0x7f59b80d0770 con 0x7f59c0103400 2024-04-26T23:04:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.727+0000 7f59bf7fe640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f59a80a3c40 0x7f59a80a6100 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009230 tx=0x7f59b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.731+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f59b8016030 con 0x7f59c0103400 2024-04-26T23:04:48.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:48.918+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f59c01a1450 con 0x7f59c0103400 2024-04-26T23:04:49.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.595+0000 7f59bcff9640 1 -- 172.21.15.1:0/2571385606 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) ==== 169+0+0 (secure 0 0 0) 0x7f59b809d010 con 0x7f59c0103400 2024-04-26T23:04:49.596 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-04-26T23:04:49.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.600+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f59a80a3c40 msgr2=0x7f59a80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:49.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.600+0000 7f59c6028640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f59a80a3c40 0x7f59a80a6100 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009230 tx=0x7f59b00314a0 comp rx=0 tx=0).stop 2024-04-26T23:04:49.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.600+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 msgr2=0x7f59c01a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:49.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.600+0000 7f59c6028640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 0x7f59c01a0130 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f59b800eaa0 tx=0x7f59b800ef60 comp rx=0 tx=0).stop 2024-04-26T23:04:49.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 shutdown_connections 2024-04-26T23:04:49.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:6800/4125558078,v1:172.21.15.1:6801/4125558078] conn(0x7f59a80a3c40 0x7f59a80a6100 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:49.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59c0102a40 0x7f59c01990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:49.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c0103400 0x7f59c01a0130 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:49.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 --2- 172.21.15.1:0/2571385606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c0101850 0x7f59c0198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:49.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 >> 172.21.15.1:0/2571385606 conn(0x7f59c00fcda0 msgr2=0x7f59c0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:49.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 shutdown_connections 2024-04-26T23:04:49.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T23:04:49.601+0000 7f59c6028640 1 -- 172.21.15.1:0/2571385606 wait complete. 2024-04-26T23:04:49.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:04:49.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:04:49.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:04:49.709 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T23:04:49.710 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T23:04:49.727 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T23:04:49.745 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T23:04:49.766 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T23:04:49.767 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T23:04:49.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a 2024-04-26T23:04:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:49 smithi001 ceph-mon[29435]: pgmap v807: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-26T23:04:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:49 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/421584002' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T23:04:49.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:49 smithi001 ceph-mon[29435]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:04:49.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:49 smithi001 ceph-mon[29435]: from='client.? 172.21.15.1:0/2571385606' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T23:04:49.847 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:49 smithi001 systemd[1]: Stopping Ceph mon.a for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:49 smithi038 ceph-mon[32433]: pgmap v807: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-26T23:04:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:49 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/421584002' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T23:04:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:49 smithi038 ceph-mon[32433]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:04:49.851 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:49 smithi038 ceph-mon[32433]: from='client.? 172.21.15.1:0/2571385606' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T23:04:49.985 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:49 smithi186 ceph-mon[32774]: pgmap v807: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-26T23:04:50.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:49 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/421584002' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T23:04:50.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:49 smithi186 ceph-mon[32774]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:04:50.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:49 smithi186 ceph-mon[32774]: from='client.? 172.21.15.1:0/2571385606' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T23:04:50.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a[29431]: 2024-04-26T23:04:50.087+0000 7f9428995700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:04:50.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a[29431]: 2024-04-26T23:04:50.087+0000 7f9428995700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T23:04:50.346 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:50 smithi001 podman[83680]: 2024-04-26 23:04:50.129562421 +0000 UTC m=+0.163812740 container died f7d0c6f1dd9ebd0dbd04eb3419a7982ca5406d03c372031a8a4b58e242c40674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:04:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:50 smithi001 podman[83680]: 2024-04-26 23:04:50.580604712 +0000 UTC m=+0.614855073 container cleanup f7d0c6f1dd9ebd0dbd04eb3419a7982ca5406d03c372031a8a4b58e242c40674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:04:50.846 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 23:04:50 smithi001 bash[83680]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-a 2024-04-26T23:04:51.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T23:04:51.045 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:04:51.045 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T23:04:51.046 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T23:04:51.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.b 2024-04-26T23:04:51.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:51 smithi038 systemd[1]: Stopping Ceph mon.b for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:51.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b[32429]: 2024-04-26T23:04:51.376+0000 7fee5a038700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:04:51.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b[32429]: 2024-04-26T23:04:51.376+0000 7fee5a038700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T23:04:51.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:51 smithi038 podman[66754]: 2024-04-26 23:04:51.500632136 +0000 UTC m=+0.248561606 container died 714e571c3bf19cd348a06a6687b78c646b17659f9d457bb17b2f8cd65a3b660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T23:04:52.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:51 smithi038 podman[66754]: 2024-04-26 23:04:51.986804813 +0000 UTC m=+0.734734315 container cleanup 714e571c3bf19cd348a06a6687b78c646b17659f9d457bb17b2f8cd65a3b660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T23:04:52.101 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 23:04:51 smithi038 bash[66754]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-b 2024-04-26T23:04:52.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T23:04:52.460 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:04:52.460 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T23:04:52.460 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T23:04:52.461 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.c 2024-04-26T23:04:52.793 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:52 smithi186 systemd[1]: Stopping Ceph mon.c for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:53.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-c[32754]: 2024-04-26T23:04:52.793+0000 7f9e2df81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:04:53.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-c[32754]: 2024-04-26T23:04:52.793+0000 7f9e2df81700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T23:04:53.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:52 smithi186 podman[67542]: 2024-04-26 23:04:52.834822446 +0000 UTC m=+0.167310363 container died 4d2a25e14b360a36b88ce295e6fb394fdeaa224f52f8c935e427b7543adec279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T23:04:53.374 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:53 smithi186 podman[67542]: 2024-04-26 23:04:53.353856951 +0000 UTC m=+0.686344907 container cleanup 4d2a25e14b360a36b88ce295e6fb394fdeaa224f52f8c935e427b7543adec279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T23:04:53.374 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:53 smithi186 bash[67542]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-c 2024-04-26T23:04:53.831 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T23:04:53.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:53 smithi186 podman[67554]: 2024-04-26 23:04:53.596440304 +0000 UTC m=+0.760999488 container remove 4d2a25e14b360a36b88ce295e6fb394fdeaa224f52f8c935e427b7543adec279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T23:04:53.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:53 smithi186 systemd[1]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-26T23:04:53.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:53 smithi186 systemd[1]: Stopped Ceph mon.c for bb8b501a-041d-11ef-bc93-c7b262605968. 2024-04-26T23:04:53.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 23:04:53 smithi186 systemd[1]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mon.c.service: Consumed 12.729s CPU time. 2024-04-26T23:04:53.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:04:53.879 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T23:04:53.880 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T23:04:53.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x 2024-04-26T23:04:54.193 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 23:04:53 smithi001 systemd[1]: Stopping Ceph mgr.x for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:54.494 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 23:04:54 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T23:04:54.192+0000 7fd48f0e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:04:54.494 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 23:04:54 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x[29668]: 2024-04-26T23:04:54.192+0000 7fd48f0e7700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T23:04:54.494 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 23:04:54 smithi001 podman[83802]: 2024-04-26 23:04:54.224759391 +0000 UTC m=+0.151916811 container died 2b3a40c98d9a07fb5f9f9b8d295ead2c4fd58d3f1f17451ca98dd73e0bbfeaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T23:04:54.846 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 23:04:54 smithi001 podman[83802]: 2024-04-26 23:04:54.619361821 +0000 UTC m=+0.546519252 container cleanup 2b3a40c98d9a07fb5f9f9b8d295ead2c4fd58d3f1f17451ca98dd73e0bbfeaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:04:54.846 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 23:04:54 smithi001 bash[83802]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-x 2024-04-26T23:04:54.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T23:04:55.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:04:55.028 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T23:04:55.029 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T23:04:55.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.y 2024-04-26T23:04:55.351 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 23:04:55 smithi038 systemd[1]: Stopping Ceph mgr.y for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:55.676 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 23:04:55 smithi038 podman[66880]: 2024-04-26 23:04:55.371470083 +0000 UTC m=+0.145633073 container died 5c25694dee7b809ac6f1d30ae7c4ae49650ad1ac389715b39e8740e09f1f8419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T23:04:56.101 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 23:04:55 smithi038 podman[66880]: 2024-04-26 23:04:55.76826102 +0000 UTC m=+0.542424007 container cleanup 5c25694dee7b809ac6f1d30ae7c4ae49650ad1ac389715b39e8740e09f1f8419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-26T23:04:56.101 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 23:04:55 smithi038 bash[66880]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-y 2024-04-26T23:04:56.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T23:04:56.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:04:56.398 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T23:04:56.399 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T23:04:56.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.z 2024-04-26T23:04:56.728 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 23:04:56 smithi186 systemd[1]: Stopping Ceph mgr.z for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:56.985 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 23:04:56 smithi186 podman[67664]: 2024-04-26 23:04:56.729076077 +0000 UTC m=+0.139129862 container died 61802ca1077d9687a96a72edb5895eb0f166764a962d82522076fa3720b4fff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:04:57.376 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 23:04:57 smithi186 podman[67664]: 2024-04-26 23:04:57.117345521 +0000 UTC m=+0.527399310 container cleanup 61802ca1077d9687a96a72edb5895eb0f166764a962d82522076fa3720b4fff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:04:57.376 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 23:04:57 smithi186 bash[67664]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mgr-z 2024-04-26T23:04:57.492 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T23:04:57.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:04:57.525 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T23:04:57.525 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T23:04:57.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.0 2024-04-26T23:04:57.841 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:04:57 smithi001 systemd[1]: Stopping Ceph osd.0 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:04:58.096 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:04:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0[42090]: 2024-04-26T23:04:57.839+0000 7fee0fbe0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:04:58.096 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:04:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0[42090]: 2024-04-26T23:04:57.839+0000 7fee0fbe0700 -1 osd.0 90 *** Got signal Terminated *** 2024-04-26T23:04:58.096 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:04:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0[42090]: 2024-04-26T23:04:57.839+0000 7fee0fbe0700 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:03.475 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:03 smithi001 podman[83934]: 2024-04-26 23:05:03.178444817 +0000 UTC m=+5.458731404 container died ddfa7eed59e6c8a33e8a2ce0a50941e39291ae3ade152c40ccad7c613488bd0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T23:05:03.817 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:03 smithi001 podman[83934]: 2024-04-26 23:05:03.599510526 +0000 UTC m=+5.879797103 container cleanup ddfa7eed59e6c8a33e8a2ce0a50941e39291ae3ade152c40ccad7c613488bd0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-26T23:05:03.817 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:03 smithi001 bash[83934]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0 2024-04-26T23:05:04.096 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:03 smithi001 podman[83953]: 2024-04-26 23:05:03.817721078 +0000 UTC m=+0.632011721 container remove ddfa7eed59e6c8a33e8a2ce0a50941e39291ae3ade152c40ccad7c613488bd0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T23:05:04.596 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:04 smithi001 podman[84044]: 2024-04-26 23:05:04.142673791 +0000 UTC m=+0.018368562 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:04.596 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:04 smithi001 podman[84044]: 2024-04-26 23:05:04.327155268 +0000 UTC m=+0.202850072 container create 90e6aacf292722a5407014f85cd545722c5af925e1e85c02a32c65fbbf0be9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T23:05:04.596 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:04 smithi001 podman[84044]: 2024-04-26 23:05:04.596634123 +0000 UTC m=+0.472328886 container init 90e6aacf292722a5407014f85cd545722c5af925e1e85c02a32c65fbbf0be9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T23:05:04.846 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:04 smithi001 podman[84044]: 2024-04-26 23:05:04.599974825 +0000 UTC m=+0.475669641 container start 90e6aacf292722a5407014f85cd545722c5af925e1e85c02a32c65fbbf0be9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T23:05:04.846 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:04 smithi001 podman[84044]: 2024-04-26 23:05:04.704980928 +0000 UTC m=+0.580675693 container attach 90e6aacf292722a5407014f85cd545722c5af925e1e85c02a32c65fbbf0be9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T23:05:05.216 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 23:05:04 smithi001 podman[84044]: 2024-04-26 23:05:04.905732835 +0000 UTC m=+0.781427604 container died 90e6aacf292722a5407014f85cd545722c5af925e1e85c02a32c65fbbf0be9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:05:05.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T23:05:05.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:05.603 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T23:05:05.604 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T23:05:05.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.1 2024-04-26T23:05:05.922 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:05 smithi038 systemd[1]: Stopping Ceph osd.1 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:06.351 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1[37012]: 2024-04-26T23:05:05.921+0000 7f1e02a75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:06.351 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1[37012]: 2024-04-26T23:05:05.921+0000 7f1e02a75700 -1 osd.1 90 *** Got signal Terminated *** 2024-04-26T23:05:06.352 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1[37012]: 2024-04-26T23:05:05.921+0000 7f1e02a75700 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:11.774 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:11 smithi038 podman[67016]: 2024-04-26 23:05:11.464686235 +0000 UTC m=+5.666623689 container died b0d461925e44fbecd2298f61fe775ce73bb68bcc37985ef0a967e8d6314f4195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T23:05:12.099 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:11 smithi038 podman[67016]: 2024-04-26 23:05:11.873591362 +0000 UTC m=+6.075528795 container cleanup b0d461925e44fbecd2298f61fe775ce73bb68bcc37985ef0a967e8d6314f4195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-26T23:05:12.100 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:11 smithi038 bash[67016]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1 2024-04-26T23:05:12.351 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:12 smithi038 podman[67035]: 2024-04-26 23:05:12.10023358 +0000 UTC m=+0.637118703 container remove b0d461925e44fbecd2298f61fe775ce73bb68bcc37985ef0a967e8d6314f4195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T23:05:12.851 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:12 smithi038 podman[67126]: 2024-04-26 23:05:12.413752271 +0000 UTC m=+0.018481868 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:12.851 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:12 smithi038 podman[67126]: 2024-04-26 23:05:12.526532114 +0000 UTC m=+0.131281732 container create 45d11693bba045d4328abefe0787fccf6b3264203a2404545a7ea30fed196f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T23:05:13.279 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:12 smithi038 podman[67126]: 2024-04-26 23:05:12.970824495 +0000 UTC m=+0.575554080 container init 45d11693bba045d4328abefe0787fccf6b3264203a2404545a7ea30fed196f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:05:13.279 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:12 smithi038 podman[67126]: 2024-04-26 23:05:12.974351242 +0000 UTC m=+0.579080834 container start 45d11693bba045d4328abefe0787fccf6b3264203a2404545a7ea30fed196f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T23:05:13.279 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:13 smithi038 podman[67126]: 2024-04-26 23:05:13.054177748 +0000 UTC m=+0.658907343 container attach 45d11693bba045d4328abefe0787fccf6b3264203a2404545a7ea30fed196f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T23:05:13.601 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 23:05:13 smithi038 podman[67126]: 2024-04-26 23:05:13.280110684 +0000 UTC m=+0.884840271 container died 45d11693bba045d4328abefe0787fccf6b3264203a2404545a7ea30fed196f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T23:05:13.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T23:05:14.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:14.015 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T23:05:14.015 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T23:05:14.016 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.2 2024-04-26T23:05:14.312 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:14 smithi186 systemd[1]: Stopping Ceph osd.2 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:14.735 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2[37918]: 2024-04-26T23:05:14.312+0000 7f70f19b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:14.735 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2[37918]: 2024-04-26T23:05:14.312+0000 7f70f19b6700 -1 osd.2 90 *** Got signal Terminated *** 2024-04-26T23:05:14.735 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2[37918]: 2024-04-26T23:05:14.312+0000 7f70f19b6700 -1 osd.2 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:19.947 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:19 smithi186 podman[67804]: 2024-04-26 23:05:19.663776823 +0000 UTC m=+5.458716407 container died 1a142d53f7d21e3671d2c1376cfcc1c924741b58ae1f04fe07d180e4f708d0ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T23:05:20.235 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67804]: 2024-04-26 23:05:20.03836379 +0000 UTC m=+5.833303372 container cleanup 1a142d53f7d21e3671d2c1376cfcc1c924741b58ae1f04fe07d180e4f708d0ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:05:20.235 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 bash[67804]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2 2024-04-26T23:05:20.613 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67821]: 2024-04-26 23:05:20.28181839 +0000 UTC m=+0.614703831 container remove 1a142d53f7d21e3671d2c1376cfcc1c924741b58ae1f04fe07d180e4f708d0ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T23:05:20.893 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67917]: 2024-04-26 23:05:20.517078085 +0000 UTC m=+0.018200325 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:20.893 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67917]: 2024-04-26 23:05:20.649412918 +0000 UTC m=+0.150535146 container create 3e69755374b10fab33f17589008b7f03e76e1bcae6f08684179091d43a1c422e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-26T23:05:21.196 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67917]: 2024-04-26 23:05:20.89408547 +0000 UTC m=+0.395207698 container init 3e69755374b10fab33f17589008b7f03e76e1bcae6f08684179091d43a1c422e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T23:05:21.196 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67917]: 2024-04-26 23:05:20.897494252 +0000 UTC m=+0.398616491 container start 3e69755374b10fab33f17589008b7f03e76e1bcae6f08684179091d43a1c422e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-26T23:05:21.196 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:20 smithi186 podman[67917]: 2024-04-26 23:05:20.995162494 +0000 UTC m=+0.496284723 container attach 3e69755374b10fab33f17589008b7f03e76e1bcae6f08684179091d43a1c422e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5) 2024-04-26T23:05:21.463 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 23:05:21 smithi186 podman[67917]: 2024-04-26 23:05:21.196787951 +0000 UTC m=+0.697910184 container died 3e69755374b10fab33f17589008b7f03e76e1bcae6f08684179091d43a1c422e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T23:05:21.846 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T23:05:21.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:21.884 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T23:05:21.884 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T23:05:21.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.3 2024-04-26T23:05:22.186 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:21 smithi001 systemd[1]: Stopping Ceph osd.3 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:22.596 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T23:05:22.186+0000 7fca702c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:22.596 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T23:05:22.186+0000 7fca702c5700 -1 osd.3 90 *** Got signal Terminated *** 2024-04-26T23:05:22.596 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:22 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T23:05:22.186+0000 7fca702c5700 -1 osd.3 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:24.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:24.098+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:25.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:24 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:24.732+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:25.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:25.131+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:25.596 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T23:05:25.307+0000 7fca6c8b9700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:01.507169+0000 front 2024-04-26T23:05:01.507219+0000 (oldest deadline 2024-04-26T23:05:25.007305+0000) 2024-04-26T23:05:26.095 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:25 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:25.724+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:26.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:25 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:25.728+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:26.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:26.166+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:26.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:26.265+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:26.596 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:26 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3[49068]: 2024-04-26T23:05:26.261+0000 7fca6c8b9700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:01.507169+0000 front 2024-04-26T23:05:01.507219+0000 (oldest deadline 2024-04-26T23:05:25.007305+0000) 2024-04-26T23:05:26.880 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:26 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:26.773+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:27.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:26 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:26.771+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:27.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:27.250+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:27.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:27.142+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:27.995 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:27 smithi001 podman[84156]: 2024-04-26 23:05:27.690597648 +0000 UTC m=+5.610154074 container died b3c71b7dcdb2232428b63eb73bba05dc137c0d8d1999bb532ab5f058fc51b2ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:05:27.995 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:27 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:27.793+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:28.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:27 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:27.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:28.262 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:28.089+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:28.262 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 podman[84156]: 2024-04-26 23:05:28.078382187 +0000 UTC m=+5.997938617 container cleanup b3c71b7dcdb2232428b63eb73bba05dc137c0d8d1999bb532ab5f058fc51b2ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-26T23:05:28.263 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 bash[84156]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3 2024-04-26T23:05:28.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:28.314+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:28.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:28.159+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:28.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:28.234+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:28.595 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 podman[84185]: 2024-04-26 23:05:28.262974737 +0000 UTC m=+0.569619960 container remove b3c71b7dcdb2232428b63eb73bba05dc137c0d8d1999bb532ab5f058fc51b2ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T23:05:28.958 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:28 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:28.749+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:28.958 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 podman[84277]: 2024-04-26 23:05:28.60728162 +0000 UTC m=+0.018089023 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:28.958 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 podman[84277]: 2024-04-26 23:05:28.722669788 +0000 UTC m=+0.133477202 container create 4640379fa36bf15f733097a2f84e757719dbe498574ab1043e95326727ada743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:05:28.959 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 podman[84277]: 2024-04-26 23:05:28.958641235 +0000 UTC m=+0.369448606 container init 4640379fa36bf15f733097a2f84e757719dbe498574ab1043e95326727ada743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T23:05:29.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:28 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:28.792+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:29.262 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:29 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:29.125+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:29.262 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:28 smithi001 podman[84277]: 2024-04-26 23:05:28.962091074 +0000 UTC m=+0.372898447 container start 4640379fa36bf15f733097a2f84e757719dbe498574ab1043e95326727ada743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T23:05:29.263 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:29 smithi001 podman[84277]: 2024-04-26 23:05:29.183788994 +0000 UTC m=+0.594596389 container attach 4640379fa36bf15f733097a2f84e757719dbe498574ab1043e95326727ada743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:05:29.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:29.324+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:29.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:29.139+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:29.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:29.213+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:29.561 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 23:05:29 smithi001 podman[84277]: 2024-04-26 23:05:29.26386796 +0000 UTC m=+0.674675350 container died 4640379fa36bf15f733097a2f84e757719dbe498574ab1043e95326727ada743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T23:05:29.846 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:29 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:29.757+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:29.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T23:05:29.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:29.937 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T23:05:29.938 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T23:05:29.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.4 2024-04-26T23:05:30.101 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:29 smithi038 systemd[1]: Stopping Ceph osd.4 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:30.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:29 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:29.786+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:30.345 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:30 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:30.125+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:30.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:30.326+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:30.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:30.157+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:30.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:30.253+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:30.601 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:30 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4[43545]: 2024-04-26T23:05:30.261+0000 7ff7c232f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:30.601 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:30 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4[43545]: 2024-04-26T23:05:30.261+0000 7ff7c232f700 -1 osd.4 90 *** Got signal Terminated *** 2024-04-26T23:05:30.601 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:30 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4[43545]: 2024-04-26T23:05:30.261+0000 7ff7c232f700 -1 osd.4 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:31.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:30 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:30.784+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:31.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:30 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:30.806+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:31.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:31 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:31.345+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:31.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:31 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:31.138+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:31.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:31 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:31.291+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:31.595 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:31 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:31.110+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:32.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:31 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:31.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:32.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:31 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:31.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:32.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:31 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:31.785+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:32.468 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:32 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:32.364+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:32.469 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:32 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:32.176+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:32.469 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:32 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:32.340+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:32.596 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:32 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:32.112+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:33.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:32 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:32.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:33.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:32 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:32.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:33.154 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:32 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:32.853+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:33.155 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:32 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:32.853+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:33.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:33 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:33.392+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:33.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:33 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:33.165+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:33.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:33 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:33.310+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:33.595 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:33 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:33.155+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:34.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:33 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:33.793+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:34.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:33 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:33.793+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:34.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:34.139+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:34.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:34.139+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:34.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:33 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:33.899+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:34.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:33 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:33.899+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:34.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:34 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:34.209+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:34.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:34 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:34.358+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:34.486 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:34 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:34.358+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:34.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:34 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:34.301+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:34.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:34 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:34.301+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:35.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:34 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:34.803+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:35.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:34 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:34.803+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:35.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:35 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:35.022+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:35.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:35.100+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:35.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:35.100+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:35.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:34.932+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:35.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:34.932+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:35.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:35 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:35.327+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:35.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:35 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:35.327+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:35.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:35 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:35.168+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:35.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:35 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:35.168+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:35.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:35 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:35.295+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:35.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:35 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:35.295+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:35.970 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:35 smithi038 podman[67232]: 2024-04-26 23:05:35.696483531 +0000 UTC m=+5.558545109 container died 0f375bd15d7bb1d454ea1126472e3f90a377be489182e03dfb9c6d82fd6e386a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:05:35.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:35 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:35.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:35.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:35 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:35.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:36.321 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 podman[67232]: 2024-04-26 23:05:36.070180721 +0000 UTC m=+5.932242303 container cleanup 0f375bd15d7bb1d454ea1126472e3f90a377be489182e03dfb9c6d82fd6e386a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:05:36.321 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 bash[67232]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4 2024-04-26T23:05:36.321 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:35 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:35.992+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:36.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:36.087+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:36.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:36.087+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:36.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:35.965+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:36.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:35 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:35.965+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:36.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:36 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:36.283+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:36.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:36 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:36.283+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:36.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:36 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:36.307+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:36.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:36 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:36.307+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:36.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:36 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:36.167+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:36.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:36 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:36.167+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:36.601 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 podman[67249]: 2024-04-26 23:05:36.321403287 +0000 UTC m=+0.621409662 container remove 0f375bd15d7bb1d454ea1126472e3f90a377be489182e03dfb9c6d82fd6e386a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T23:05:36.915 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:36 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:36.775+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:36.916 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:36 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:36.775+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:36.916 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 podman[67341]: 2024-04-26 23:05:36.655616108 +0000 UTC m=+0.114753056 container create fd96a45693b4f98278a4299dad55cd2946e5895b2bd228182b751d900af3bcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T23:05:36.916 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 podman[67341]: 2024-04-26 23:05:36.559288889 +0000 UTC m=+0.018425843 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:36.916 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 podman[67341]: 2024-04-26 23:05:36.916684062 +0000 UTC m=+0.375821012 container init fd96a45693b4f98278a4299dad55cd2946e5895b2bd228182b751d900af3bcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:05:37.223 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:37 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:37.014+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:37.224 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:36 smithi038 podman[67341]: 2024-04-26 23:05:36.919828326 +0000 UTC m=+0.378965275 container start fd96a45693b4f98278a4299dad55cd2946e5895b2bd228182b751d900af3bcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T23:05:37.224 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:37 smithi038 podman[67341]: 2024-04-26 23:05:37.048645672 +0000 UTC m=+0.507782644 container attach fd96a45693b4f98278a4299dad55cd2946e5895b2bd228182b751d900af3bcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T23:05:37.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:37 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:37.047+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:37.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:37 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:37.047+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:37.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:36.991+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:37.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:36 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:36.991+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:37.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:37 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:37.270+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:37.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:37 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:37.270+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:37.486 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:37 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:37.301+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:37.486 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:37 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:37.301+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:37.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:37 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:37.183+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:37.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:37 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:37.183+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:37.595 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 23:05:37 smithi038 podman[67341]: 2024-04-26 23:05:37.224467583 +0000 UTC m=+0.683604529 container died fd96a45693b4f98278a4299dad55cd2946e5895b2bd228182b751d900af3bcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:05:37.851 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:37 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:37.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:37.851 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:37 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:37.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:37.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T23:05:37.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:37.952 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T23:05:37.952 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T23:05:37.953 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.5 2024-04-26T23:05:38.235 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:37 smithi186 systemd[1]: Stopping Ceph osd.5 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:38.235 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:38.208+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:38.235 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:38.208+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:38.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:38.050+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:38.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:38.050+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:38.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:37 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:37.970+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:38.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:37 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:37.970+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:38.351 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:38 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:38.010+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:38.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:38.252+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:38.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:38.252+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:38.735 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:38.277+0000 7f6d6dab6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:38.735 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:38.277+0000 7f6d6dab6700 -1 osd.5 90 *** Got signal Terminated *** 2024-04-26T23:05:38.735 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:38.277+0000 7f6d6dab6700 -1 osd.5 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:38.736 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:38.290+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:38.736 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:38 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:38.290+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:39.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:38 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:38.986+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:39.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:38 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:38.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:39.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:38 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:38.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:39.335 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:38.920+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:39.336 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:38 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:38.920+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:39.336 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:39.027+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:39.336 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:39.027+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:39.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:39.275+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:39.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:39.275+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:39.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:39.317+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:39.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:39.317+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:39.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:39.206+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:39.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:39 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:39.206+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:40.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:39 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:39.962+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:40.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:39 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:39.765+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:40.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:39 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:39.765+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:40.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:39.885+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:40.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:39.885+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:40.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:40.063+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:40.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:40.063+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:40.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:40.063+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:40.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:40.243+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:40.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:40.243+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:40.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:40.361+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:40.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:40.361+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:40.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:40.191+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:40.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:40 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:40.191+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:41.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:40 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:40.911+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:41.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:40 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:40.753+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:41.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:40 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:40.753+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:41.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:40.893+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:41.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:40 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:40.893+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:41.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:41.052+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:41.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:41.052+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:41.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:41.052+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:41.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:41.230+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:41.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:41.230+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:41.485 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:41.369+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:41.486 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:41.369+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:41.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:41.231+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:41.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:41.231+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:41.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:41 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:41.231+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:42.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:41 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:41.877+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:42.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:41 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:41.745+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:42.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:41 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:41.745+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:42.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:41 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:41.745+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:42.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:42 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:42.088+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:42.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:42 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:42.088+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:42.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:42 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:42.088+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:42.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:41.925+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:42.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:41.925+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:42.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:41.925+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:42.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:42.260+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:42.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:42.260+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:42.735 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:42.331+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.391625+0000 front 2024-04-26T23:05:02.391495+0000 (oldest deadline 2024-04-26T23:05:27.691484+0000) 2024-04-26T23:05:42.736 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5[44421]: 2024-04-26T23:05:42.331+0000 7f6d660a2700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.692379+0000 front 2024-04-26T23:05:07.692222+0000 (oldest deadline 2024-04-26T23:05:33.592053+0000) 2024-04-26T23:05:42.736 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:42.281+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:42.736 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:42.281+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:42.736 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:42 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:42.281+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:43.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:42.860+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:43.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:42.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:43.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:42.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:43.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:42 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:42.791+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:43.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:42 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:42.950+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:43.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:42 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:42.950+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:43.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:42 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:42.950+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:43.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:43.087+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:43.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:43.087+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:43.347 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:43.087+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:43.718 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:43.238+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:43.718 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:43.238+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:43.719 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:43.238+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:43.719 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:43.295+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:43.719 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:43.295+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:43.719 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:43 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:43.295+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:43.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:43 smithi186 podman[68049]: 2024-04-26 23:05:43.719152076 +0000 UTC m=+5.573366038 container died 7234b1243646a5043a8ba36231059cdbbd914cdf50ee4e46c9b1dbaa8ad3112b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T23:05:44.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:43.885+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:44.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:43.885+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:44.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:43.798+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:44.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:43.798+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:44.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:43 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:43.798+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:44.262 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 podman[68049]: 2024-04-26 23:05:44.062168991 +0000 UTC m=+5.916382960 container cleanup 7234b1243646a5043a8ba36231059cdbbd914cdf50ee4e46c9b1dbaa8ad3112b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T23:05:44.263 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 bash[68049]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5 2024-04-26T23:05:44.263 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:44.195+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:44.263 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:44.196+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:44.263 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:44.196+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:44.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:44 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:44.102+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:44.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:44 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:44.102+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:44.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:44 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:44.102+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:44.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:43.969+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:44.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:43.969+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:44.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:43.969+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:44.680 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 podman[68067]: 2024-04-26 23:05:44.263453209 +0000 UTC m=+0.542782391 container remove 7234b1243646a5043a8ba36231059cdbbd914cdf50ee4e46c9b1dbaa8ad3112b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T23:05:44.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:44.312+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:44.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:44.312+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:44.681 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:44 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:44.312+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:44.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 podman[68160]: 2024-04-26 23:05:44.681051546 +0000 UTC m=+0.113236139 container create da90dcf27d2aa1360a8328049de1e33b49cfff872216d61c0bb5b4d7398b3bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T23:05:44.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 podman[68160]: 2024-04-26 23:05:44.586558013 +0000 UTC m=+0.018742628 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:44.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 podman[68160]: 2024-04-26 23:05:44.909363206 +0000 UTC m=+0.341547801 container init da90dcf27d2aa1360a8328049de1e33b49cfff872216d61c0bb5b4d7398b3bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-26T23:05:44.985 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:44 smithi186 podman[68160]: 2024-04-26 23:05:44.912591114 +0000 UTC m=+0.344775709 container start da90dcf27d2aa1360a8328049de1e33b49cfff872216d61c0bb5b4d7398b3bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T23:05:45.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:44.848+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:45.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:44.848+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:45.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:44.816+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:45.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:44.816+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:45.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:44 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:44.816+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:45.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:45 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:45.100+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:45.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:45 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:45.100+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:45.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:45 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:45.100+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:45.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:44 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:44.998+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:45.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:44 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:44.998+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:45.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:44 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:44.998+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:45.362 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:45 smithi186 podman[68160]: 2024-04-26 23:05:45.009387588 +0000 UTC m=+0.441572184 container attach da90dcf27d2aa1360a8328049de1e33b49cfff872216d61c0bb5b4d7398b3bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:05:45.363 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 23:05:45 smithi186 podman[68160]: 2024-04-26 23:05:45.2056114 +0000 UTC m=+0.637796000 container died da90dcf27d2aa1360a8328049de1e33b49cfff872216d61c0bb5b4d7398b3bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T23:05:45.363 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:45.162+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:45.363 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:45.162+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:45.363 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:45.162+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:45.652 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:45.362+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:45.653 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:45.362+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:45.653 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:45 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:45.362+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:45.878 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T23:05:45.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:45.915 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T23:05:45.916 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T23:05:45.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.6 2024-04-26T23:05:46.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:45.815+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:46.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:45.815+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:46.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:45.783+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:46.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:45.783+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:46.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:45 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:45.783+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:46.232 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:46.002+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:46.232 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:46.002+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:46.232 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:46.002+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:46.232 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:45 smithi001 systemd[1]: Stopping Ceph osd.6 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:46.232 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:46.071+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:46.233 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:46.071+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:46.233 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:46.071+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:46.233 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:46.231+0000 7f088e7f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:46.233 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:46.231+0000 7f088e7f8700 -1 osd.6 90 *** Got signal Terminated *** 2024-04-26T23:05:46.233 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:46 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:46.231+0000 7f088e7f8700 -1 osd.6 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:46.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:46.359+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:46.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:46.359+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:46.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:46.359+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:46.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:46.191+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:46.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:46.191+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:46.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:46 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:46.191+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:47.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:46.821+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:47.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:46.821+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:47.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:46.800+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:47.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:46.800+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:47.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:46.800+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:47.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:47 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:47.032+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:47.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:47 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:47.032+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:47.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:47 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:47.032+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:47.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:47 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:47.007+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:47.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:47 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:47.007+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:47.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:47 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:47.007+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:47.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:47.185+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:47.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:47.185+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:47.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:47.185+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:47.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:47.345+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:47.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:47.345+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:47.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:47 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:47.345+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:48.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:47.787+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:48.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:47.787+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:48.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:47.787+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:48.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:47.860+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:48.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:47 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:47.860+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:48.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:48 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:48.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:48.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:48 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:48.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:48.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:48 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:48.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:48.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:48 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:48.053+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:48.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:48 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:48.053+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:48.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:48 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:48.053+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:48.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:48.157+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:48.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:48.157+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:48.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:48.157+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:48.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:48.157+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:48.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:48.331+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:48.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:48.331+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:48.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:48 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:48.331+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:49.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:48.819+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:49.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:48.819+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:49.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:48.746+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:49.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:48.746+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:49.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:48 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:48.746+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:49.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:49 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:49.125+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:49.433 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:49 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:49.125+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:49.433 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:49 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:49.125+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:49.433 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:49 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:49.083+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:49.433 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:49 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:49.083+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:49.433 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:49 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:49.083+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:49.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:49.339+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:49.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:49.339+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:49.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:49.339+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:49.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:49.117+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:49.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:49.117+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:49.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:49.117+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:49.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:49 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:49.117+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:50.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:49.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:50.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:49.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:50.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:49.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:50.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:49.824+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:50.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:49.824+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:50.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:49 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:49.824+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:50.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:50.084+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:50.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:50.084+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:50.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:50.084+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:50.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:50.084+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.285823+0000 front 2024-04-26T23:05:25.285847+0000 (oldest deadline 2024-04-26T23:05:49.986043+0000) 2024-04-26T23:05:50.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:50.074+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:50.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:50.074+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:50.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:50.074+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:50.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:50 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:50.074+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:50.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:50.330+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:50.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:50.330+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:50.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:50.330+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:50.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:50.103+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:50.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:50.103+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:50.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:50.103+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:50.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:50 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:50.103+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:51.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:50 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:50.778+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:51.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:50 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:50.778+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:51.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:50 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:50.778+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:51.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:50 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:50.782+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:51.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:50 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:50.782+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:51.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:50 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:50.782+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:51.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:51.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.181154+0000 front 2024-04-26T23:05:00.181186+0000 (oldest deadline 2024-04-26T23:05:27.782312+0000) 2024-04-26T23:05:51.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:51.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.283682+0000 front 2024-04-26T23:05:08.283789+0000 (oldest deadline 2024-04-26T23:05:33.583632+0000) 2024-04-26T23:05:51.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:51.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.084689+0000 front 2024-04-26T23:05:17.084746+0000 (oldest deadline 2024-04-26T23:05:39.984809+0000) 2024-04-26T23:05:51.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6[55900]: 2024-04-26T23:05:51.079+0000 7f0886de4700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.285823+0000 front 2024-04-26T23:05:25.285847+0000 (oldest deadline 2024-04-26T23:05:49.986043+0000) 2024-04-26T23:05:51.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.034+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:51.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.034+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:51.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.034+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:51.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.034+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:51.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:51.322+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:51.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:51.322+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:51.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:51.322+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:51.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:51.084+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:51.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:51.084+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:51.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:51.084+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:51.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:51 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:51.084+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:51.916 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:51 smithi001 podman[84381]: 2024-04-26 23:05:51.645982009 +0000 UTC m=+5.532799017 container died 906ce795d063007e79a19d89f7550a60937fce28d43dc40e59dbe03a08ffa089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T23:05:52.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:51.749+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:52.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:51.749+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:52.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:51.749+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:52.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:51.775+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:52.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:51.775+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:52.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:51.775+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:52.265 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84381]: 2024-04-26 23:05:52.015385468 +0000 UTC m=+5.902202464 container cleanup 906ce795d063007e79a19d89f7550a60937fce28d43dc40e59dbe03a08ffa089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:05:52.266 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 bash[84381]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6 2024-04-26T23:05:52.266 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.992+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:52.266 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.992+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:52.266 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.992+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:52.267 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:51 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:51.992+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:52.344 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:52.091+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:52.344 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:52.091+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:52.344 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:52.091+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:52.344 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:52.091+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:52.583 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84399]: 2024-04-26 23:05:52.266326607 +0000 UTC m=+0.613853625 container remove 906ce795d063007e79a19d89f7550a60937fce28d43dc40e59dbe03a08ffa089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T23:05:52.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:52.344+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:52.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:52.344+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:52.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:52 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:52.344+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:52.846 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84493]: 2024-04-26 23:05:52.583874057 +0000 UTC m=+0.109665704 container create 125f8af32ecc86d2f24166099f50dd34508821974df0e24be7dede0f74de4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T23:05:52.846 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84493]: 2024-04-26 23:05:52.492924103 +0000 UTC m=+0.018715736 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:05:52.846 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84493]: 2024-04-26 23:05:52.794655203 +0000 UTC m=+0.320446809 container init 125f8af32ecc86d2f24166099f50dd34508821974df0e24be7dede0f74de4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-26T23:05:52.847 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84493]: 2024-04-26 23:05:52.797996999 +0000 UTC m=+0.323788605 container start 125f8af32ecc86d2f24166099f50dd34508821974df0e24be7dede0f74de4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T23:05:53.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:52.735+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:53.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:52.735+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:53.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:52.735+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:53.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:52.759+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:53.103 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:52.759+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:53.103 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:52.759+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:53.103 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:52 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:52.759+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:53.339 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:53.078+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:53.339 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:53.078+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:53.339 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:53.078+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:53.339 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:53.078+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:53.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:52 smithi001 podman[84493]: 2024-04-26 23:05:52.894934846 +0000 UTC m=+0.420726461 container attach 125f8af32ecc86d2f24166099f50dd34508821974df0e24be7dede0f74de4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T23:05:53.346 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 23:05:53 smithi001 podman[84493]: 2024-04-26 23:05:53.105876485 +0000 UTC m=+0.631668103 container died 125f8af32ecc86d2f24166099f50dd34508821974df0e24be7dede0f74de4815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:05:53.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.000+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:53.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.000+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:53.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.000+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:53.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.000+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:53.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:53.339+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:53.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:53.339+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:53.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:53 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:53.339+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:53.889 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T23:05:53.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:05:53.936 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T23:05:53.936 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T23:05:53.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.7 2024-04-26T23:05:53.985 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:53.731+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:53.986 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:53.731+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:53.986 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:53.731+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:53.986 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:53.716+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:53.986 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:53.716+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:53.987 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:53.716+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:53.987 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:53 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:53.716+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:54.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.980+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:54.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.980+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:54.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.980+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:54.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:53 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:53.980+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:54.351 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:53 smithi038 systemd[1]: Stopping Ceph osd.7 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:05:54.381 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:54.075+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:54.381 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:54.075+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:54.381 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:54.075+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:54.381 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:54.075+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:54.697 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:54.356+0000 7f137dccf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:05:54.697 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:54.356+0000 7f137dccf700 -1 osd.7 90 *** Got signal Terminated *** 2024-04-26T23:05:54.697 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:54.356+0000 7f137dccf700 -1 osd.7 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:05:54.697 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:54.690+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:54.697 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:54.690+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:54.697 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:54.690+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:54.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:54.381+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:54.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:54.381+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:54.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:54 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:54.381+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:55.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:54.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:55.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:54.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:55.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:54.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:55.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:54 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:54.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:55.251 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:54 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:54.944+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:55.251 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:54 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:54.944+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:55.251 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:54 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:54.944+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:55.252 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:54 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:54.944+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:55.377 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:55.048+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:55.378 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:55.048+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:55.378 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:55.048+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:55.378 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:55.048+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:55.737 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:55.376+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:55.738 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:55.376+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:55.738 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:55 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:55.376+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:56.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:55.689+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:56.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:55.689+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:56.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:55.689+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:56.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:55.705+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:56.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:55.705+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:56.103 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:55.705+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:56.103 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:55.705+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:56.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:55.955+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:56.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:55.955+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:56.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:55.955+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:56.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:55 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:55.955+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:56.367 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:56.032+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:56.367 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:56.032+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:56.368 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:56.033+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:56.368 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:56.033+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:56.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:56.368+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:56.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:56.368+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:56.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:56 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:56.368+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:57.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:56.686+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:57.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:56.686+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:57.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:56.686+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:57.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:56.726+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:57.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:56.726+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:57.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:56.726+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:57.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:56 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:56.726+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:57.335 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:57.041+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:57.335 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:57.041+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:57.335 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:57.041+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:57.335 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:57.041+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:57.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:56 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:56.948+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:57.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:56 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:56.948+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:57.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:56 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:56.948+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:57.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:56 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:56.948+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:57.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:57.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:57.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:57.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:57.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:57.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:57.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:57 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:57.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:05:58.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:57.736+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:58.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:57.736+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:58.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:57.736+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:58.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:57.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:58.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:57.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:58.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:57.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:58.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:57 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:57.754+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:58.334 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:58.053+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:58.334 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:58.053+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:58.334 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:58.053+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:58.335 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:58.053+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:05:58.345 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:57.911+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:58.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:57.911+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:58.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:57.911+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:58.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:57 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:57.911+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:58.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:58.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:58.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:58.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:58.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:58.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:58.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:58 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:58.335+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:05:59.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:58.790+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:05:59.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:58.790+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:05:59.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:58.790+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:05:59.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:58.790+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:05:59.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:58.688+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.881022+0000 front 2024-04-26T23:05:08.881062+0000 (oldest deadline 2024-04-26T23:05:34.181481+0000) 2024-04-26T23:05:59.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:58.688+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.683649+0000 front 2024-04-26T23:05:18.683789+0000 (oldest deadline 2024-04-26T23:05:43.383484+0000) 2024-04-26T23:05:59.102 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:58 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7[50251]: 2024-04-26T23:05:58.688+0000 7f137a2c3700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.485495+0000 front 2024-04-26T23:05:25.485542+0000 (oldest deadline 2024-04-26T23:05:49.585434+0000) 2024-04-26T23:05:59.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:58.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:05:59.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:58.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:05:59.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:58.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:05:59.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:58 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:58.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:05:59.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:59.320+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:05:59.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:59.320+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:05:59.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:59.320+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:05:59.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:05:59.320+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:05:59.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:59.096+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:05:59.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:59.096+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:05:59.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:59.096+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:05:59.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:05:59 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:05:59.096+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:00.032 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:05:59 smithi038 podman[67445]: 2024-04-26 23:05:59.754665358 +0000 UTC m=+5.618820727 container died bdae9158c5c3e57dbf1fe51ccaaa3d7e3f460c75f049c34f7d8cd7d5605c9bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:00.032 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:59 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:59.755+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:00.033 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:59 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:59.755+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:00.033 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:59 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:59.755+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:00.033 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:05:59 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:05:59.755+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:00.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:59.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:00.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:59.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:00.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:59.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:00.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:59.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:00.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:05:59 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:05:59.912+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:00.351 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:00 smithi038 podman[67445]: 2024-04-26 23:06:00.131774548 +0000 UTC m=+5.995929910 container cleanup bdae9158c5c3e57dbf1fe51ccaaa3d7e3f460c75f049c34f7d8cd7d5605c9bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T23:06:00.351 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:00 smithi038 bash[67445]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7 2024-04-26T23:06:00.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:00.319+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:00.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:00.319+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:00.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:00.319+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:00.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:00.319+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:00.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:00.098+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:00.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:00.098+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:00.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:00.098+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:00.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:00 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:00.098+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:00.716 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:00 smithi038 podman[67462]: 2024-04-26 23:06:00.400124725 +0000 UTC m=+0.646083544 container remove bdae9158c5c3e57dbf1fe51ccaaa3d7e3f460c75f049c34f7d8cd7d5605c9bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T23:06:01.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:00 smithi038 podman[67554]: 2024-04-26 23:06:00.619736249 +0000 UTC m=+0.018291970 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:06:01.101 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:00 smithi038 podman[67554]: 2024-04-26 23:06:00.742606722 +0000 UTC m=+0.141162426 container create a8b9b6ecd543fff27ae83cf1358d674cf45cdf223f3e8732bddb2fdd923c4618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:06:01.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:00 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:00.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:01.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:00 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:00.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:01.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:00 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:00.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:01.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:00 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:00.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:01.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:00.874+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:01.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:00.874+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:01.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:00.874+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:01.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:00.874+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:01.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:00 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:00.874+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:01.384 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:01 smithi038 podman[67554]: 2024-04-26 23:06:01.119250378 +0000 UTC m=+0.517806091 container init a8b9b6ecd543fff27ae83cf1358d674cf45cdf223f3e8732bddb2fdd923c4618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T23:06:01.384 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:01 smithi038 podman[67554]: 2024-04-26 23:06:01.122890357 +0000 UTC m=+0.521446079 container start a8b9b6ecd543fff27ae83cf1358d674cf45cdf223f3e8732bddb2fdd923c4618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T23:06:01.385 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:01 smithi038 podman[67554]: 2024-04-26 23:06:01.17862076 +0000 UTC m=+0.577176471 container attach a8b9b6ecd543fff27ae83cf1358d674cf45cdf223f3e8732bddb2fdd923c4618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:01.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:01.131+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:01.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:01.131+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:01.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:01.131+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:01.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:01.131+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:01.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:01.287+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:01.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:01.287+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:01.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:01.287+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:01.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:01 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:01.287+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:01.722 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 23:06:01 smithi038 podman[67554]: 2024-04-26 23:06:01.439070159 +0000 UTC m=+0.837625857 container died a8b9b6ecd543fff27ae83cf1358d674cf45cdf223f3e8732bddb2fdd923c4618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:06:02.048 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:01 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:01.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:02.048 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:01 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:01.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:02.048 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:01 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:01.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:02.049 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:01 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:01.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:02.049 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:01 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:01.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:02.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T23:06:02.095 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:01.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:02.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:01.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:02.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:01.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:02.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:01.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:02.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:01 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:01.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:02.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:02.101 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T23:06:02.101 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T23:06:02.102 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.8 2024-04-26T23:06:02.425 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:02.245+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:02.425 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:02.245+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:02.425 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:02.245+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:02.426 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:02.245+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:02.426 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.135+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:02.426 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.135+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:02.426 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.135+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:02.426 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.135+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:02.427 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 systemd[1]: Stopping Ceph osd.8 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:02.427 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.425+0000 7fd8b3fdf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:02.427 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.425+0000 7fd8b3fdf700 -1 osd.8 90 *** Got signal Terminated *** 2024-04-26T23:06:02.427 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:02 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:02.425+0000 7fd8b3fdf700 -1 osd.8 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:06:03.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:02 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:02.693+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:03.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:02 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:02.693+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:03.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:02 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:02.693+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:03.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:02 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:02.693+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:03.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:02 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:02.693+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:03.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:02.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:03.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:02.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:03.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:02.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:03.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:02.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:03.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:02 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:02.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:03.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:03.177+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:03.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:03.177+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:03.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:03.177+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:03.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:03.177+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:03.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:03.285+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:03.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:03.285+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:03.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:03.285+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:03.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:03.285+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:04.003 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:03 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:03.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:04.003 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:03 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:03.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:04.003 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:03 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:03.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:04.004 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:03 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:03.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:04.004 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:03 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:03.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:04.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:03 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:03.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:04.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:03 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:03.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:04.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:03 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:03.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:04.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:03 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:03.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:04.346 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:03 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:03.852+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:04.387 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:04.136+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:04.388 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:04.136+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:04.388 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:04.136+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:04.388 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:04.136+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:04.388 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:04.308+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:04.388 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:04.308+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:04.389 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:04.308+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:04.389 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:04.308+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:04.389 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:04 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:04.308+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:05.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:04.836+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:05.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:04.836+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:05.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:04.836+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:05.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:04.836+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:05.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:04 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:04.836+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:05.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:04 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:04.661+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:05.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:04 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:04.661+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:05.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:04 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:04.661+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:05.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:04 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:04.661+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:05.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:04 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:04.661+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:05.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:05.099+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:05.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:05.099+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:05.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:05.099+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:05.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:05.099+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:05.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:05.099+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.236617+0000 front 2024-04-26T23:05:41.236639+0000 (oldest deadline 2024-04-26T23:06:04.736908+0000) 2024-04-26T23:06:05.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:05.343+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:05.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:05.343+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:05.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:05.343+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:05.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:05.343+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:05.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:05 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:05.343+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:06.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:05 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:05.797+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:06.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:05 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:05.797+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:06.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:05 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:05.797+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:06.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:05 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:05.797+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:06.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:05 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:05.797+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:06.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:05.709+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:06.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:05.709+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:06.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:05.709+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:06.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:05.709+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:06.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:05.709+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:06.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:05 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:05.709+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:06.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:06.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:06.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:06.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:06.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:06.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:06.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:06.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:06.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:06.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.236617+0000 front 2024-04-26T23:05:41.236639+0000 (oldest deadline 2024-04-26T23:06:04.736908+0000) 2024-04-26T23:06:06.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:06.328+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:06.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:06.328+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:06.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:06.328+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:06.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:06.328+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:06.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:06 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:06.328+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:07.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:06 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:06.844+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:07.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:06 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:06.844+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:07.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:06 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:06.844+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:07.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:06 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:06.844+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:07.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:06 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:06.844+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:07.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:06 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:06.844+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:07.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:06.710+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:07.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:06.710+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:07.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:06.710+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:07.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:06.710+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:07.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:06.710+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:07.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:06 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:06.710+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:07.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:07.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:04:57.931216+0000 front 2024-04-26T23:04:57.931095+0000 (oldest deadline 2024-04-26T23:05:23.831244+0000) 2024-04-26T23:06:07.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:07.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:09.132514+0000 front 2024-04-26T23:05:09.132388+0000 (oldest deadline 2024-04-26T23:05:34.432428+0000) 2024-04-26T23:06:07.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:07.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.533354+0000 front 2024-04-26T23:05:18.533378+0000 (oldest deadline 2024-04-26T23:05:40.233556+0000) 2024-04-26T23:06:07.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:07.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.534496+0000 front 2024-04-26T23:05:25.534686+0000 (oldest deadline 2024-04-26T23:05:47.834465+0000) 2024-04-26T23:06:07.486 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8[50960]: 2024-04-26T23:06:07.134+0000 7fd8ac5cb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.236617+0000 front 2024-04-26T23:05:41.236639+0000 (oldest deadline 2024-04-26T23:06:04.736908+0000) 2024-04-26T23:06:07.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:07.375+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:07.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:07.375+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:07.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:07.375+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:07.487 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:07.375+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:07.487 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:07.375+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:08.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:07.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:08.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:07.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:08.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:07.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:08.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:07.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:08.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:07.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:08.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:07 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:07.832+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:08.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:07 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:07.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:08.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:07 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:07.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:08.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:07 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:07.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:08.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:07 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:07.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:08.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:07 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:07.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:08.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:07 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:07.751+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:08.160 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:07 smithi186 podman[68267]: 2024-04-26 23:06:07.881439469 +0000 UTC m=+5.586489497 container died 515ac8cccd69f99f1db756f5ff650b3de7c5dcca4238c64d9e1ea718474c85cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T23:06:08.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:08 smithi186 podman[68267]: 2024-04-26 23:06:08.259115229 +0000 UTC m=+5.964165256 container cleanup 515ac8cccd69f99f1db756f5ff650b3de7c5dcca4238c64d9e1ea718474c85cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T23:06:08.485 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:08 smithi186 bash[68267]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8 2024-04-26T23:06:08.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:08.346+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:08.485 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:08.346+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:08.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:08.346+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:08.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:08.346+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:08.486 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:08 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:08.346+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:08.876 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:08 smithi186 podman[68296]: 2024-04-26 23:06:08.535469364 +0000 UTC m=+0.653374283 container remove 515ac8cccd69f99f1db756f5ff650b3de7c5dcca4238c64d9e1ea718474c85cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T23:06:09.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:08 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:08.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:09.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:08 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:08.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:09.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:08 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:08.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:09.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:08 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:08.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:09.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:08 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:08.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:09.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:08 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:08.825+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:09.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:08 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:08.711+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:09.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:08 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:08.711+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:09.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:08 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:08.711+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:09.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:08 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:08.711+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:09.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:08 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:08.711+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:09.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:08 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:08.711+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:09.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:08 smithi186 podman[68390]: 2024-04-26 23:06:08.780587877 +0000 UTC m=+0.018085874 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:06:09.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:08 smithi186 podman[68390]: 2024-04-26 23:06:08.886138025 +0000 UTC m=+0.123636037 container create a44d7ab4c9c1578c329e6c487a78d158b94c07a11d7124db497fd7d449a51961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:06:09.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:09 smithi186 podman[68390]: 2024-04-26 23:06:09.180865743 +0000 UTC m=+0.418363710 container init a44d7ab4c9c1578c329e6c487a78d158b94c07a11d7124db497fd7d449a51961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:06:09.477 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:09 smithi186 podman[68390]: 2024-04-26 23:06:09.184062096 +0000 UTC m=+0.421560062 container start a44d7ab4c9c1578c329e6c487a78d158b94c07a11d7124db497fd7d449a51961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:09.477 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:09 smithi186 podman[68390]: 2024-04-26 23:06:09.289108682 +0000 UTC m=+0.526606648 container attach a44d7ab4c9c1578c329e6c487a78d158b94c07a11d7124db497fd7d449a51961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-26T23:06:09.477 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:09 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:09.377+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:09.478 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:09 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:09.377+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:09.478 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:09 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:09.377+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:09.478 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:09 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:09.377+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:09.478 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:09 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:09.377+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:09.735 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 23:06:09 smithi186 podman[68390]: 2024-04-26 23:06:09.477802898 +0000 UTC m=+0.715300875 container died a44d7ab4c9c1578c329e6c487a78d158b94c07a11d7124db497fd7d449a51961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T23:06:10.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:09.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:10.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:09.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:10.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:09.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:10.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:09.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:10.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:09.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:10.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:09 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:09.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:10.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:09 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:09.720+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:10.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:09 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:09.720+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:10.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:09 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:09.720+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:10.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:09 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:09.720+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:10.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:09 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:09.720+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:10.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:09 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:09.720+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:10.167 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T23:06:10.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:10.206 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T23:06:10.207 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T23:06:10.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.9 2024-04-26T23:06:10.525 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 systemd[1]: Stopping Ceph osd.9 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:10.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:10 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:10.402+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:10.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:10 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:10.402+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:10.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:10 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:10.402+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:10.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:10 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:10.402+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:10.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:10 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:10.402+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:10.846 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.525+0000 7fe615105700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:10.846 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.525+0000 7fe615105700 -1 osd.9 90 *** Got signal Terminated *** 2024-04-26T23:06:10.846 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.525+0000 7fe615105700 -1 osd.9 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:06:10.847 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:10.847 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:10.847 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:10.847 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:10.847 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:10.847 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:10 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:10.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:11.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:10 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:10.684+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:11.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:10 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:10.684+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:11.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:10 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:10.684+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:11.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:10 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:10.684+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:11.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:10 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:10.684+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:11.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:10 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:10.684+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:11.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:11.440+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:11.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:11.440+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:11.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:11.440+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:11.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:11.440+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:11.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:11.440+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:12.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:11 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:11.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:12.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:11 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:11.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:12.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:11 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:11.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:12.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:11 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:11.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:12.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:11 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:11.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:12.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:11 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:11.760+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:12.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:11 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:11.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:12.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:11 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:11.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:12.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:11 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:11.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:12.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:11 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:11.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:12.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:11 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:11.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:12.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:11 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:11.668+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:12.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:12 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:12.433+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:12.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:12 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:12.433+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:12.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:12 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:12.433+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:12.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:12 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:12.433+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:12.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:12 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:12.433+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:13.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:12.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:13.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:12.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:13.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:12.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:13.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:12.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:13.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:12.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:13.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:12 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:12.803+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:13.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:12 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:12.714+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:13.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:12 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:12.714+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:13.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:12 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:12.714+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:13.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:12 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:12.714+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:13.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:12 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:12.714+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:13.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:12 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:12.714+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:13.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:13 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:13.449+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:13.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:13 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:13.449+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:13.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:13 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:13.449+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:13.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:13 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:13.449+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:13.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:13 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:13.449+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:14.059 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:13 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:13.760+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:14.059 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:13 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:13.760+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:14.059 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:13 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:13.760+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:14.059 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:13 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:13.760+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:14.059 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:13 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:13.760+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:14.060 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:13 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:13.760+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:14.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:14.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:14.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:14.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:14.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:14.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:14.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:13 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:13.795+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:48.896753+0000 front 2024-04-26T23:05:48.896729+0000 (oldest deadline 2024-04-26T23:06:13.596801+0000) 2024-04-26T23:06:14.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:14.494+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:14.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:14.494+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:14.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:14.494+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:14.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:14.494+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:14.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:14 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:14.494+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:15.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.088244+0000 front 2024-04-26T23:05:00.088388+0000 (oldest deadline 2024-04-26T23:05:24.788382+0000) 2024-04-26T23:06:15.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.889697+0000 front 2024-04-26T23:05:08.889678+0000 (oldest deadline 2024-04-26T23:05:31.690132+0000) 2024-04-26T23:06:15.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.491656+0000 front 2024-04-26T23:05:17.491788+0000 (oldest deadline 2024-04-26T23:05:40.991615+0000) 2024-04-26T23:06:15.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:26.192988+0000 front 2024-04-26T23:05:26.192941+0000 (oldest deadline 2024-04-26T23:05:49.093060+0000) 2024-04-26T23:06:15.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.394075+0000 front 2024-04-26T23:05:34.394264+0000 (oldest deadline 2024-04-26T23:05:59.694088+0000) 2024-04-26T23:06:15.096 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:43.195188+0000 front 2024-04-26T23:05:43.195232+0000 (oldest deadline 2024-04-26T23:06:06.595763+0000) 2024-04-26T23:06:15.097 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:14 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9[62506]: 2024-04-26T23:06:14.831+0000 7fe6116f9700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:48.896753+0000 front 2024-04-26T23:05:48.896729+0000 (oldest deadline 2024-04-26T23:06:13.596801+0000) 2024-04-26T23:06:15.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:15.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:15.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:15.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:15.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:15.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:15.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:14 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:14.719+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:15.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:15.499+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:15.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:15.499+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:15.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:15.499+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:15.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:15.499+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:15.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:15.499+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:16.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:16.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:16.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:16.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:16.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:16.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:16.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:15 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:15.696+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:16.319 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:16 smithi001 podman[84611]: 2024-04-26 23:06:16.057689977 +0000 UTC m=+5.651448936 container died 1b4fcf60fef500bcae49c07408a24e1c6bff75315ed61b5d37199928c715674a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T23:06:16.596 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:16 smithi001 podman[84611]: 2024-04-26 23:06:16.409159125 +0000 UTC m=+6.002918082 container cleanup 1b4fcf60fef500bcae49c07408a24e1c6bff75315ed61b5d37199928c715674a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-26T23:06:16.596 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:16 smithi001 bash[84611]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9 2024-04-26T23:06:16.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:16 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:16.475+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:16.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:16 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:16.475+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:16.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:16 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:16.475+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:16.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:16 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:16.475+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:16.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:16 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:16.475+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:16.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:16 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:16.475+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:17.077 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:16 smithi001 podman[84628]: 2024-04-26 23:06:16.718652036 +0000 UTC m=+0.656722300 container remove 1b4fcf60fef500bcae49c07408a24e1c6bff75315ed61b5d37199928c715674a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T23:06:17.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:17.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:17.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:17.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:17.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:17.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:17.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:16 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:16.704+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:17.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:17 smithi001 podman[84719]: 2024-04-26 23:06:17.077751725 +0000 UTC m=+0.110557756 container create 6ba656382ed54b7d2c12b0c57e715e568e347444a57153b69b10e343f16e0afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:06:17.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:17 smithi001 podman[84719]: 2024-04-26 23:06:16.985483179 +0000 UTC m=+0.018289214 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:06:17.347 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:17 smithi001 podman[84719]: 2024-04-26 23:06:17.280161239 +0000 UTC m=+0.312967262 container init 6ba656382ed54b7d2c12b0c57e715e568e347444a57153b69b10e343f16e0afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T23:06:17.348 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:17 smithi001 podman[84719]: 2024-04-26 23:06:17.283447554 +0000 UTC m=+0.316253577 container start 6ba656382ed54b7d2c12b0c57e715e568e347444a57153b69b10e343f16e0afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:17.846 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:17 smithi001 podman[84719]: 2024-04-26 23:06:17.346716305 +0000 UTC m=+0.379522344 container attach 6ba656382ed54b7d2c12b0c57e715e568e347444a57153b69b10e343f16e0afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T23:06:17.846 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 23:06:17 smithi001 podman[84719]: 2024-04-26 23:06:17.589085766 +0000 UTC m=+0.621891795 container died 6ba656382ed54b7d2c12b0c57e715e568e347444a57153b69b10e343f16e0afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T23:06:17.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:17 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:17.485+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:17.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:17 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:17.485+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:17.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:17 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:17.485+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:17.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:17 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:17.485+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:17.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:17 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:17.485+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:17.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:17 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:17.485+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:18.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:18.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:18.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:18.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:18.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:18.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:18.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:17 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:17.681+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:18.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T23:06:18.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:18.215 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T23:06:18.216 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T23:06:18.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.10 2024-04-26T23:06:18.527 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 systemd[1]: Stopping Ceph osd.10 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:18.851 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.527+0000 7f51427ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:18.851 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.527+0000 7f51427ad700 -1 osd.10 90 *** Got signal Terminated *** 2024-04-26T23:06:18.851 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.527+0000 7f51427ad700 -1 osd.10 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:18.852 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:18 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:18.721+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:18.988 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:18.534+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:18.988 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:18.534+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:18.988 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:18.534+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:18.988 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:18.534+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:18.988 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:18.534+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:18.989 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:18 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:18.534+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:19.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:19 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:19.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:19.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:19 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:19.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:19.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:19 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:19.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:19.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:19 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:19.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:19.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:19 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:19.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:19.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:19 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:19.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:20.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:20.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:20.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:20.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:20.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:20.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:20.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:19 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:19.729+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:20.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:20 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:20.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:20.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:20 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:20.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:20.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:20 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:20.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:20.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:20 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:20.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:20.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:20 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:20.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:20.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:20 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:20.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:21.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:21.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:21.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:21.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:21.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:21.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:21.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:20 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:20.723+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:21.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:21.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:21.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:21.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:21.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:21.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:21.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:21 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:21.503+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:22.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:22.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:22.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:22.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:22.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:22.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:22.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:21 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:21.749+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:22.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:22.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:22.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:22.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:22.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:22.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:22.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:22 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:22.525+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:23.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:02.750373+0000 front 2024-04-26T23:05:02.750397+0000 (oldest deadline 2024-04-26T23:05:23.850204+0000) 2024-04-26T23:06:23.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:07.951070+0000 front 2024-04-26T23:05:07.951101+0000 (oldest deadline 2024-04-26T23:05:32.651264+0000) 2024-04-26T23:06:23.101 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:18.353388+0000 front 2024-04-26T23:05:18.353344+0000 (oldest deadline 2024-04-26T23:05:41.253384+0000) 2024-04-26T23:06:23.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T23:05:25.954668+0000 front 2024-04-26T23:05:25.954490+0000 (oldest deadline 2024-04-26T23:05:51.854482+0000) 2024-04-26T23:06:23.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:31.854878+0000 front 2024-04-26T23:05:31.854917+0000 (oldest deadline 2024-04-26T23:06:01.255492+0000) 2024-04-26T23:06:23.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.256511+0000 front 2024-04-26T23:05:41.256377+0000 (oldest deadline 2024-04-26T23:06:04.756217+0000) 2024-04-26T23:06:23.102 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:22 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10[56848]: 2024-04-26T23:06:22.763+0000 7f513eda1700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.557803+0000 front 2024-04-26T23:05:50.557689+0000 (oldest deadline 2024-04-26T23:06:14.657621+0000) 2024-04-26T23:06:23.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:23.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:23.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:23.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:23.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:23.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:23.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:23 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:23.480+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:24.103 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:23 smithi038 podman[67669]: 2024-04-26 23:06:23.84294204 +0000 UTC m=+5.420846951 container died c79a8992d11b097dc84d8ffad842ae5a871a4173f48e8704e5b21afffcceff66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T23:06:24.477 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:24 smithi038 podman[67669]: 2024-04-26 23:06:24.226727931 +0000 UTC m=+5.804632853 container cleanup c79a8992d11b097dc84d8ffad842ae5a871a4173f48e8704e5b21afffcceff66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T23:06:24.477 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:24 smithi038 bash[67669]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10 2024-04-26T23:06:24.783 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:24 smithi038 podman[67687]: 2024-04-26 23:06:24.477835005 +0000 UTC m=+0.624605116 container remove c79a8992d11b097dc84d8ffad842ae5a871a4173f48e8704e5b21afffcceff66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:06:24.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:24.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:24.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:24.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:24.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:24.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:24.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:24 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:24.512+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:25.033 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:24 smithi038 podman[67780]: 2024-04-26 23:06:24.687101396 +0000 UTC m=+0.018259507 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:06:25.034 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:24 smithi038 podman[67780]: 2024-04-26 23:06:24.820356045 +0000 UTC m=+0.151514155 container create 9ae7220f9ef7689b771146a85fde23dc21f8a6d9c586453d10a4266259df7d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424) 2024-04-26T23:06:25.034 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:25 smithi038 podman[67780]: 2024-04-26 23:06:25.031467133 +0000 UTC m=+0.362625244 container init 9ae7220f9ef7689b771146a85fde23dc21f8a6d9c586453d10a4266259df7d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:25.034 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:25 smithi038 podman[67780]: 2024-04-26 23:06:25.034573577 +0000 UTC m=+0.365731694 container start 9ae7220f9ef7689b771146a85fde23dc21f8a6d9c586453d10a4266259df7d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T23:06:25.337 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:25 smithi038 podman[67780]: 2024-04-26 23:06:25.114703844 +0000 UTC m=+0.445861955 container attach 9ae7220f9ef7689b771146a85fde23dc21f8a6d9c586453d10a4266259df7d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:25.601 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 23:06:25 smithi038 podman[67780]: 2024-04-26 23:06:25.338144259 +0000 UTC m=+0.669302373 container died 9ae7220f9ef7689b771146a85fde23dc21f8a6d9c586453d10a4266259df7d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T23:06:25.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T23:06:25.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:25.954 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T23:06:25.955 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T23:06:25.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.11 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:25.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:25 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:25.533+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:26.287 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 systemd[1]: Stopping Ceph osd.11 for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:26.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.287+0000 7ffbe0a74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:26.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.287+0000 7ffbe0a74700 -1 osd.11 90 *** Got signal Terminated *** 2024-04-26T23:06:26.735 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.287+0000 7ffbe0a74700 -1 osd.11 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:26.736 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:26 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:26.506+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:27.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:27.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:27.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:27.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:27.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:27.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:27.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:27 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:27.520+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:28.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:28.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:28.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:28.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:28.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:28.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:28.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:28 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:28.550+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:29.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:29.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:29.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:29.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:29.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:29.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:29.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:29 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:29.524+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:30.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T23:05:00.971453+0000 front 2024-04-26T23:05:00.971484+0000 (oldest deadline 2024-04-26T23:05:25.671275+0000) 2024-04-26T23:06:30.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T23:05:08.572279+0000 front 2024-04-26T23:05:08.572379+0000 (oldest deadline 2024-04-26T23:05:33.872283+0000) 2024-04-26T23:06:30.985 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T23:05:17.274207+0000 front 2024-04-26T23:05:17.274046+0000 (oldest deadline 2024-04-26T23:05:43.174237+0000) 2024-04-26T23:06:30.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T23:05:34.176864+0000 front 2024-04-26T23:05:34.176788+0000 (oldest deadline 2024-04-26T23:05:56.476770+0000) 2024-04-26T23:06:30.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T23:05:41.177766+0000 front 2024-04-26T23:05:41.177734+0000 (oldest deadline 2024-04-26T23:06:04.077916+0000) 2024-04-26T23:06:30.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T23:05:50.479240+0000 front 2024-04-26T23:05:50.479303+0000 (oldest deadline 2024-04-26T23:06:15.779131+0000) 2024-04-26T23:06:30.986 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:30 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11[57869]: 2024-04-26T23:06:30.515+0000 7ffbd9060700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T23:05:55.779571+0000 front 2024-04-26T23:05:55.779655+0000 (oldest deadline 2024-04-26T23:06:21.079597+0000) 2024-04-26T23:06:31.963 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:31 smithi186 podman[68495]: 2024-04-26 23:06:31.603719923 +0000 UTC m=+5.444697406 container died 5ebcd9356c6ebfcf2156069d20c4393d2d3dfc71bd67cffbae5219cf554e9482 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T23:06:32.235 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68495]: 2024-04-26 23:06:32.037392844 +0000 UTC m=+5.878370309 container cleanup 5ebcd9356c6ebfcf2156069d20c4393d2d3dfc71bd67cffbae5219cf554e9482 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-26T23:06:32.235 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 bash[68495]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11 2024-04-26T23:06:32.655 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68512]: 2024-04-26 23:06:32.238854011 +0000 UTC m=+0.622628334 container remove 5ebcd9356c6ebfcf2156069d20c4393d2d3dfc71bd67cffbae5219cf554e9482 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:06:32.974 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68605]: 2024-04-26 23:06:32.656267394 +0000 UTC m=+0.107059253 container create 1e708ddcc903662d2eed6a05313f2b2e6221d735327995378f01ade97e5b28bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T23:06:32.975 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68605]: 2024-04-26 23:06:32.567028595 +0000 UTC m=+0.017820468 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:06:32.975 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68605]: 2024-04-26 23:06:32.892099226 +0000 UTC m=+0.342891086 container init 1e708ddcc903662d2eed6a05313f2b2e6221d735327995378f01ade97e5b28bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:06:32.975 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68605]: 2024-04-26 23:06:32.895161814 +0000 UTC m=+0.345953684 container start 1e708ddcc903662d2eed6a05313f2b2e6221d735327995378f01ade97e5b28bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T23:06:32.975 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:32 smithi186 podman[68605]: 2024-04-26 23:06:32.975193486 +0000 UTC m=+0.425985359 container attach 1e708ddcc903662d2eed6a05313f2b2e6221d735327995378f01ade97e5b28bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:06:33.235 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:33 smithi186 podman[68605]: 2024-04-26 23:06:33.191806934 +0000 UTC m=+0.642598795 container died 1e708ddcc903662d2eed6a05313f2b2e6221d735327995378f01ade97e5b28bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T23:06:33.754 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 23:06:33 smithi186 podman[68631]: 2024-04-26 23:06:33.728495077 +0000 UTC m=+0.531911033 container remove 1e708ddcc903662d2eed6a05313f2b2e6221d735327995378f01ade97e5b28bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:06:33.759 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T23:06:33.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:33.836 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T23:06:33.836 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T23:06:33.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.a 2024-04-26T23:06:34.144 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:06:33 smithi001 systemd[1]: Stopping Ceph mds.a for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:34.596 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:06:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a[69459]: 2024-04-26T23:06:34.143+0000 7fbe9f229700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:34.596 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:06:34 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a[69459]: 2024-04-26T23:06:34.143+0000 7fbe9f229700 -1 mds.a *** got signal Terminated *** 2024-04-26T23:06:38.494 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:06:38 smithi001 podman[84841]: 2024-04-26 23:06:38.15853016 +0000 UTC m=+4.123554182 container died 06e44256bb9331e0e3563df61fb9fd96508d4f91c0d6663c7c92c3b198438356 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:06:38.835 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:06:38 smithi001 podman[84841]: 2024-04-26 23:06:38.551614204 +0000 UTC m=+4.516638215 container cleanup 06e44256bb9331e0e3563df61fb9fd96508d4f91c0d6663c7c92c3b198438356 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T23:06:38.835 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 23:06:38 smithi001 bash[84841]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-a 2024-04-26T23:06:38.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T23:06:38.992 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:38.992 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T23:06:38.992 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T23:06:38.992 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.d 2024-04-26T23:06:39.096 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:39 smithi001 systemd[1]: Stopping Ceph mds.d for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:39.596 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d[68506]: 2024-04-26T23:06:39.305+0000 7fa5923c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:39.596 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:39 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d[68506]: 2024-04-26T23:06:39.305+0000 7fa5923c7700 -1 mds.d *** got signal Terminated *** 2024-04-26T23:06:40.921 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:40 smithi001 podman[84964]: 2024-04-26 23:06:40.625272491 +0000 UTC m=+1.402482985 container stop 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T23:06:40.921 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:40 smithi001 podman[84964]: 2024-04-26 23:06:40.650866367 +0000 UTC m=+1.428076862 container died 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T23:06:40.921 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:40 smithi001 podman[84964]: 2024-04-26 23:06:40.921528898 +0000 UTC m=+1.698739393 container cleanup 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T23:06:41.252 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:40 smithi001 bash[84964]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d 2024-04-26T23:06:41.252 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 23:06:41 smithi001 podman[84978]: 2024-04-26 23:06:41.148064719 +0000 UTC m=+0.512457593 container remove 6662f60e411077d77210a02e2ce9a1342b5c0432338536123bb301dadf3a3d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:06:41.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T23:06:41.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:41.304 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T23:06:41.304 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T23:06:41.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.g 2024-04-26T23:06:41.596 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:06:41 smithi001 systemd[1]: Stopping Ceph mds.g for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:42.096 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:06:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g[69114]: 2024-04-26T23:06:41.633+0000 7fcc081bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:42.096 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:06:41 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g[69114]: 2024-04-26T23:06:41.633+0000 7fcc081bf700 -1 mds.g *** got signal Terminated *** 2024-04-26T23:06:42.934 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:06:42 smithi001 podman[85085]: 2024-04-26 23:06:42.640098782 +0000 UTC m=+1.113418546 container died e60acb5583e374d30443fbbdb4a4ac6e3225a22213ca60634a4a712113a35b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:06:43.267 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:06:43 smithi001 podman[85085]: 2024-04-26 23:06:43.033707536 +0000 UTC m=+1.507027308 container cleanup e60acb5583e374d30443fbbdb4a4ac6e3225a22213ca60634a4a712113a35b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:06:43.268 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 23:06:43 smithi001 bash[85085]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-g 2024-04-26T23:06:43.385 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T23:06:43.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:43.417 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T23:06:43.417 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T23:06:43.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.j 2024-04-26T23:06:43.596 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:43 smithi001 systemd[1]: Stopping Ceph mds.j for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:44.096 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j[68155]: 2024-04-26T23:06:43.745+0000 7fc68f397700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:44.096 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:43 smithi001 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j[68155]: 2024-04-26T23:06:43.745+0000 7fc68f397700 -1 mds.j *** got signal Terminated *** 2024-04-26T23:06:46.029 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:45 smithi001 podman[85206]: 2024-04-26 23:06:45.779230898 +0000 UTC m=+2.140914680 container died 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-26T23:06:46.296 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:46 smithi001 podman[85206]: 2024-04-26 23:06:46.103813544 +0000 UTC m=+2.465497316 container cleanup 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T23:06:46.296 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:46 smithi001 bash[85206]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j 2024-04-26T23:06:46.547 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 23:06:46 smithi001 podman[85220]: 2024-04-26 23:06:46.296545304 +0000 UTC m=+0.515990236 container remove 06280d7ead9b4715d46b06ccb58ef2b91f74e093478e66ba6c6afe12d88be4e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T23:06:46.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T23:06:46.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:46.582 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T23:06:46.582 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T23:06:46.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.b 2024-04-26T23:06:46.877 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:46 smithi038 systemd[1]: Stopping Ceph mds.b for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:47.153 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b[62411]: 2024-04-26T23:06:46.876+0000 7f3e883fc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:47.153 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:46 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b[62411]: 2024-04-26T23:06:46.876+0000 7f3e883fc700 -1 mds.b *** got signal Terminated *** 2024-04-26T23:06:50.984 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:50 smithi038 podman[67885]: 2024-04-26 23:06:50.660042089 +0000 UTC m=+3.881123235 container died a479f886c44694c3a9561b983fbd6ee7859bd57d9f0457e0bbfab86b175321df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:06:50.984 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:50 smithi038 podman[67885]: 2024-04-26 23:06:50.985062889 +0000 UTC m=+4.206144037 container cleanup a479f886c44694c3a9561b983fbd6ee7859bd57d9f0457e0bbfab86b175321df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:06:51.351 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:50 smithi038 bash[67885]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b 2024-04-26T23:06:51.351 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 23:06:51 smithi038 podman[67910]: 2024-04-26 23:06:51.217955453 +0000 UTC m=+0.554790697 container remove a479f886c44694c3a9561b983fbd6ee7859bd57d9f0457e0bbfab86b175321df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T23:06:51.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T23:06:51.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:51.518 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T23:06:51.518 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T23:06:51.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.e 2024-04-26T23:06:51.822 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:51 smithi038 systemd[1]: Stopping Ceph mds.e for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:52.101 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e[61773]: 2024-04-26T23:06:51.821+0000 7f2352126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:52.101 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:51 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e[61773]: 2024-04-26T23:06:51.821+0000 7f2352126700 -1 mds.e *** got signal Terminated *** 2024-04-26T23:06:54.922 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:54 smithi038 podman[68017]: 2024-04-26 23:06:54.655046905 +0000 UTC m=+2.914840206 container died a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:06:55.339 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:55 smithi038 podman[68017]: 2024-04-26 23:06:55.005841073 +0000 UTC m=+3.265634364 container cleanup a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:06:55.340 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:55 smithi038 bash[68017]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e 2024-04-26T23:06:55.601 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 23:06:55 smithi038 podman[68031]: 2024-04-26 23:06:55.340392058 +0000 UTC m=+0.675370526 container remove a7cd8edaf247531a7ca310adacfb57aee703be8d00f23e6cdfef2c206c9081c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-26T23:06:55.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T23:06:55.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:55.640 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T23:06:55.641 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T23:06:55.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.h 2024-04-26T23:06:55.975 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:06:55 smithi038 systemd[1]: Stopping Ceph mds.h for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:56.351 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:06:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h[62082]: 2024-04-26T23:06:55.975+0000 7f75fbe63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:56.351 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:06:55 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h[62082]: 2024-04-26T23:06:55.975+0000 7f75fbe63700 -1 mds.h *** got signal Terminated *** 2024-04-26T23:06:58.951 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:06:58 smithi038 podman[68140]: 2024-04-26 23:06:58.663718362 +0000 UTC m=+2.797230886 container died da7eb814513118fd181d2f9105094cb0c1c1b09d0b4057d955c0154138115c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T23:06:59.260 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:06:59 smithi038 podman[68140]: 2024-04-26 23:06:59.051096532 +0000 UTC m=+3.184609098 container cleanup da7eb814513118fd181d2f9105094cb0c1c1b09d0b4057d955c0154138115c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-26T23:06:59.260 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 23:06:59 smithi038 bash[68140]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-h 2024-04-26T23:06:59.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T23:06:59.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:06:59.436 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T23:06:59.437 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T23:06:59.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.k 2024-04-26T23:06:59.851 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:06:59 smithi038 systemd[1]: Stopping Ceph mds.k for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:06:59.851 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:06:59 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k[61452]: 2024-04-26T23:06:59.737+0000 7f4ae1313700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:06:59.851 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:06:59 smithi038 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k[61452]: 2024-04-26T23:06:59.737+0000 7f4ae1313700 -1 mds.k *** got signal Terminated *** 2024-04-26T23:07:02.937 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:07:02 smithi038 podman[68262]: 2024-04-26 23:07:02.664810522 +0000 UTC m=+3.011465928 container died b4d7e89d01944da38811899ca801cc99754dfb851ba77a9ad27b63079af06336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T23:07:02.938 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:07:02 smithi038 podman[68262]: 2024-04-26 23:07:02.938488187 +0000 UTC m=+3.285143593 container cleanup b4d7e89d01944da38811899ca801cc99754dfb851ba77a9ad27b63079af06336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T23:07:03.230 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 23:07:02 smithi038 bash[68262]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-k 2024-04-26T23:07:03.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T23:07:03.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:07:03.406 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T23:07:03.406 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T23:07:03.406 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.c 2024-04-26T23:07:03.735 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:07:03 smithi186 systemd[1]: Stopping Ceph mds.c for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:07:04.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:07:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c[62862]: 2024-04-26T23:07:03.778+0000 7fb2a145b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:07:04.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:07:03 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c[62862]: 2024-04-26T23:07:03.778+0000 7fb2a145b700 -1 mds.c *** got signal Terminated *** 2024-04-26T23:07:06.929 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:07:06 smithi186 podman[68721]: 2024-04-26 23:07:06.647708777 +0000 UTC m=+3.043288249 container died 2c82dc6dc1c58a02e1cf1b91822ea4630603a2e3ead2780f0f501d3de4ea8e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:07:07.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:07:07 smithi186 podman[68721]: 2024-04-26 23:07:07.02827923 +0000 UTC m=+3.423858702 container cleanup 2c82dc6dc1c58a02e1cf1b91822ea4630603a2e3ead2780f0f501d3de4ea8e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T23:07:07.235 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 23:07:07 smithi186 bash[68721]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-c 2024-04-26T23:07:07.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T23:07:07.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:07:07.425 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T23:07:07.426 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T23:07:07.426 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.f 2024-04-26T23:07:07.735 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:07:07 smithi186 systemd[1]: Stopping Ceph mds.f for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:07:08.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:07:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f[62556]: 2024-04-26T23:07:07.756+0000 7f3849232700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:07:08.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:07:07 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f[62556]: 2024-04-26T23:07:07.756+0000 7f3849232700 -1 mds.f *** got signal Terminated *** 2024-04-26T23:07:10.899 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:07:10 smithi186 podman[68855]: 2024-04-26 23:07:10.647967963 +0000 UTC m=+3.003963411 container died c29e21bd6ef03c8d9305ff9934a628bd8221f5b2c57d32e38b2033199e8f678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:07:11.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:07:10 smithi186 podman[68855]: 2024-04-26 23:07:10.999168038 +0000 UTC m=+3.355163492 container cleanup c29e21bd6ef03c8d9305ff9934a628bd8221f5b2c57d32e38b2033199e8f678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:07:11.235 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 23:07:10 smithi186 bash[68855]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-f 2024-04-26T23:07:11.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T23:07:11.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:07:11.459 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T23:07:11.459 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T23:07:11.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.i 2024-04-26T23:07:11.985 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:07:11 smithi186 systemd[1]: Stopping Ceph mds.i for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:07:11.985 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:07:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T23:07:11.777+0000 7f82132dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:07:11.985 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:07:11 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i[63174]: 2024-04-26T23:07:11.777+0000 7f82132dc700 -1 mds.i *** got signal Terminated *** 2024-04-26T23:07:14.931 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:07:14 smithi186 podman[68977]: 2024-04-26 23:07:14.657258573 +0000 UTC m=+2.983096714 container died f399c051c443ab21bf378b1a82616a8793e674a2bf3e276da1d48437acfcb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T23:07:15.234 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:07:15 smithi186 podman[68977]: 2024-04-26 23:07:15.144087291 +0000 UTC m=+3.469925431 container cleanup f399c051c443ab21bf378b1a82616a8793e674a2bf3e276da1d48437acfcb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T23:07:15.235 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 23:07:15 smithi186 bash[68977]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-i 2024-04-26T23:07:15.583 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T23:07:15.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:07:15.615 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T23:07:15.615 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T23:07:15.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.l 2024-04-26T23:07:15.940 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:07:15 smithi186 systemd[1]: Stopping Ceph mds.l for bb8b501a-041d-11ef-bc93-c7b262605968... 2024-04-26T23:07:16.235 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:07:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l[62252]: 2024-04-26T23:07:15.939+0000 7f97e8f52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:07:16.235 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:07:15 smithi186 ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l[62252]: 2024-04-26T23:07:15.939+0000 7f97e8f52700 -1 mds.l *** got signal Terminated *** 2024-04-26T23:07:18.923 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:07:18 smithi186 podman[69096]: 2024-04-26 23:07:18.656669464 +0000 UTC m=+2.825473063 container died 072a0005294f43fac67b2c0e48750e032062af4aaf8c4880a6b8eda4dced9686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-26T23:07:19.235 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:07:19 smithi186 podman[69096]: 2024-04-26 23:07:19.014965501 +0000 UTC m=+3.183769086 container cleanup 072a0005294f43fac67b2c0e48750e032062af4aaf8c4880a6b8eda4dced9686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T23:07:19.235 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 23:07:19 smithi186 bash[69096]: ceph-bb8b501a-041d-11ef-bc93-c7b262605968-mds-l 2024-04-26T23:07:19.437 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bb8b501a-041d-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T23:07:19.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T23:07:19.467 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T23:07:19.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T23:07:19.667 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting cluster with fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T23:07:22.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T23:07:22.437 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T23:07:24.930 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T23:07:25.122 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T23:07:27.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:07:27.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:07:27.476 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:07:27.502 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T23:07:27.504 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi001/crash 2024-04-26T23:07:27.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash -- . 2024-04-26T23:07:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:tar: /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T23:07:27.550 INFO:teuthology.orchestra.run.smithi001.stderr:tar: Error is not recoverable: exiting now 2024-04-26T23:07:27.551 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi038/crash 2024-04-26T23:07:27.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash -- . 2024-04-26T23:07:27.626 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T23:07:27.627 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-04-26T23:07:27.629 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi186/crash 2024-04-26T23:07:27.630 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash -- . 2024-04-26T23:07:27.664 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T23:07:27.664 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-04-26T23:07:27.665 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T23:07:27.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-26T23:07:27.703 INFO:tasks.cephadm:Compressing logs... 2024-04-26T23:07:27.703 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T23:07:27.745 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T23:07:27.748 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T23:07:27.770 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log 2024-04-26T23:07:27.771 INFO:teuthology.orchestra.run.smithi001.stderr:: No such file or directory 2024-04-26T23:07:27.771 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T23:07:27.771 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T23:07:27.771 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T23:07:27.784 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T23:07:27.784 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T23:07:27.784 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 90.3% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.6.log: /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.0% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T23:07:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T23:07:27.789 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log 2024-04-26T23:07:27.789 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log: --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T23:07:27.790 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.5% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr: 93.2% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T23:07:27.791 INFO:teuthology.orchestra.run.smithi186.stderr: 82.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T23:07:27.792 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T23:07:27.794 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T23:07:27.795 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-26T23:07:27.795 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-26T23:07:27.795 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T23:07:27.796 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr: 90.5% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr: 89.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T23:07:27.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T23:07:27.798 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T23:07:27.798 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T23:07:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T23:07:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.d.log: 91.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T23:07:27.804 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T23:07:27.806 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T23:07:27.807 INFO:teuthology.orchestra.run.smithi001.stderr: 94.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T23:07:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T23:07:27.808 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T23:07:27.809 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T23:07:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T23:07:27.819 INFO:teuthology.orchestra.run.smithi186.stderr: 91.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T23:07:27.825 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T23:07:27.828 INFO:teuthology.orchestra.run.smithi186.stderr: 94.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T23:07:27.829 INFO:teuthology.orchestra.run.smithi186.stderr: 94.2% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T23:07:27.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 93.3% 94.1% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T23:07:27.835 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T23:07:27.839 INFO:teuthology.orchestra.run.smithi001.stderr: 94.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T23:07:27.840 INFO:teuthology.orchestra.run.smithi038.stderr: 94.0% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T23:07:27.850 INFO:teuthology.orchestra.run.smithi038.stderr: 94.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T23:07:27.870 INFO:teuthology.orchestra.run.smithi186.stderr: 94.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T23:07:27.884 INFO:teuthology.orchestra.run.smithi038.stderr: 94.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T23:07:27.891 INFO:teuthology.orchestra.run.smithi038.stderr: 94.5% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T23:07:27.949 INFO:teuthology.orchestra.run.smithi186.stderr: 93.7% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T23:07:28.017 INFO:teuthology.orchestra.run.smithi001.stderr: 90.4% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T23:07:28.091 INFO:teuthology.orchestra.run.smithi186.stderr: 92.2% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T23:07:28.104 INFO:teuthology.orchestra.run.smithi038.stderr: 92.0% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T23:07:28.448 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T23:07:28.471 INFO:teuthology.orchestra.run.smithi001.stderr: 92.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T23:07:28.517 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T23:07:28.545 INFO:teuthology.orchestra.run.smithi001.stderr: 92.0% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T23:07:28.566 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T23:07:28.614 INFO:teuthology.orchestra.run.smithi001.stderr: 92.6% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T23:07:28.698 INFO:teuthology.orchestra.run.smithi001.stderr: 90.3% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T23:07:28.779 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T23:07:28.808 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T23:07:28.869 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T23:07:28.871 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-26T23:07:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m1.108s 2024-04-26T23:07:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m3.779s 2024-04-26T23:07:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.155s 2024-04-26T23:07:29.023 INFO:teuthology.orchestra.run.smithi186.stderr: 92.8% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T23:07:29.037 INFO:teuthology.orchestra.run.smithi186.stderr: 92.9% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T23:07:29.039 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T23:07:29.040 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m1.278s 2024-04-26T23:07:29.040 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m5.011s 2024-04-26T23:07:29.040 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.200s 2024-04-26T23:07:29.168 INFO:teuthology.orchestra.run.smithi001.stderr: 92.9% -- replaced with /var/log/ceph/bb8b501a-041d-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T23:07:29.169 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-04-26T23:07:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m1.410s 2024-04-26T23:07:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m4.841s 2024-04-26T23:07:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.179s 2024-04-26T23:07:29.170 INFO:tasks.cephadm:Archiving logs... 2024-04-26T23:07:29.171 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi001/log 2024-04-26T23:07:29.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T23:07:29.971 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi038/log 2024-04-26T23:07:29.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T23:07:30.583 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi186/log 2024-04-26T23:07:30.584 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T23:07:31.391 INFO:tasks.cephadm:Removing cluster... 2024-04-26T23:07:31.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --force 2024-04-26T23:07:31.585 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting cluster with fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T23:07:31.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --force 2024-04-26T23:07:32.055 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T23:07:32.321 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bb8b501a-041d-11ef-bc93-c7b262605968 --force 2024-04-26T23:07:32.510 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: bb8b501a-041d-11ef-bc93-c7b262605968 2024-04-26T23:07:32.763 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T23:07:32.764 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T23:07:32.783 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T23:07:32.800 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T23:07:32.815 INFO:tasks.cephadm:Teardown complete 2024-04-26T23:07:32.815 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T23:07:32.826 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T23:07:32.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T23:07:32.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T23:07:32.842 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T23:07:32.957 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T23:07:32.967 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T23:07:32.967 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:07:32.970 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:07:32.973 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:07:32.985 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:07:32.986 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:07:32.986 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:07:32.989 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:07:32.990 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-04-26T23:07:32.990 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +1308us[+1329us] +/- 39ms 2024-04-26T23:07:32.990 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 +241us[ +262us] +/- 74ms 2024-04-26T23:07:32.990 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -3132us[-3110us] +/- 85ms 2024-04-26T23:07:32.990 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:07:32.992 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:07:32.992 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +1391us[+1383us] +/- 39ms 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 +341us[ +333us] +/- 74ms 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -3013us[-3013us] +/- 85ms 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +1389us[+1382us] +/- 39ms 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 +339us[ +339us] +/- 74ms 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -3029us[-3029us] +/- 85ms 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:07:32.994 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T23:07:33.013 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T23:07:33.013 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T23:07:33.048 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T23:07:33.127 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T23:07:33.195 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T23:07:33.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:07:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:07:33.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:07:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:07:33.318 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:07:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:07:33.345 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T23:07:33.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T23:07:33.388 INFO:teuthology.task.internal:Duration was 2090.607352 seconds 2024-04-26T23:07:33.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T23:07:33.428 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T23:07:33.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T23:07:33.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T23:07:33.434 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T23:07:33.469 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:07:33.471 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:07:33.471 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:07:33.886 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T23:07:33.887 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T23:07:33.887 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T23:07:33.911 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-26T23:07:33.911 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T23:07:33.955 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T23:07:33.955 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T23:07:33.984 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T23:07:33.984 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T23:07:33.987 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T23:07:33.997 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T23:07:34.049 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T23:07:34.049 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T23:07:34.058 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T23:07:34.065 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T23:07:34.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T23:07:34.439 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T23:07:34.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T23:07:34.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T23:07:34.489 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T23:07:34.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T23:07:34.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:07:34.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:07:34.532 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:07:34.549 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-04-26T23:07:34.553 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-26T23:07:34.577 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-26T23:07:34.650 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:07:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:07:34.664 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:07:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:07:34.681 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:07:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:07:34.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T23:07:34.715 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T23:07:34.716 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi001 2024-04-26T23:07:34.716 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T23:07:34.758 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi038 2024-04-26T23:07:34.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T23:07:34.795 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272/remote/smithi186 2024-04-26T23:07:34.795 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T23:07:34.830 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T23:07:34.830 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T23:07:34.832 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T23:07:34.835 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T23:07:34.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T23:07:34.891 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T23:07:34.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T23:07:34.911 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T23:07:34.912 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T23:07:34.914 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T23:07:34.916 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T23:07:34.927 INFO:teuthology.orchestra.run.smithi001.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 23:07 /home/ubuntu/cephtest 2024-04-26T23:07:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 23:07 /home/ubuntu/cephtest 2024-04-26T23:07:34.933 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 23:07 /home/ubuntu/cephtest 2024-04-26T23:07:34.934 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T23:07:34.944 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T23:07:35.014 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T23:07:35.045 DEBUG:teuthology.nuke:shortname: smithi001 2024-04-26T23:07:35.045 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T23:07:35.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:27:19.644092', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnCjic0Oq+Ct/r3wvi53lH5O+2bqC9vZ5P/jeGDjOmhzU/05hM2CJTwJeaz+OmcJe6HH95VqMjRNe5+9eCl9sQ='} 2024-04-26T23:07:35.077 DEBUG:teuthology.nuke:shortname: smithi038 2024-04-26T23:07:35.077 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T23:07:35.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:27:19.646609', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOb4nixO2kZ88v5vmrAbE4znWB8mC3hnrVxOJkfOERbEVQSLMGSKs4w/KFSA4dT7cA/tLdt9w+Q3OI42nt6kvns='} 2024-04-26T23:07:35.106 DEBUG:teuthology.nuke:shortname: smithi186 2024-04-26T23:07:35.106 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T23:07:35.141 INFO:teuthology.orchestra.console.smithi001:Power off 2024-04-26T23:07:35.141 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T23:07:35.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:27:19.645445', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxiJxGhunQFdo4YSRlfWv+ih6Pcpphd5HwDNkvzsTsxwgDAljeZ77XUxMCiM2XarVuznIgkvtB/C9S/NjdGwtI='} 2024-04-26T23:07:35.166 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-04-26T23:07:35.166 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:35.183 INFO:teuthology.orchestra.console.smithi038:Power off 2024-04-26T23:07:35.183 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T23:07:35.197 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-04-26T23:07:35.212 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-04-26T23:07:35.212 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:35.228 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-26T23:07:35.229 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T23:07:35.239 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-26T23:07:35.254 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-26T23:07:35.255 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:35.278 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-26T23:07:39.198 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:39.239 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:39.280 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:39.310 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-04-26T23:07:39.356 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-26T23:07:39.391 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-26T23:07:43.315 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:43.357 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:43.392 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T23:07:43.425 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-04-26T23:07:43.469 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-04-26T23:07:43.502 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-26T23:07:43.526 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-04-26T23:07:43.574 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-04-26T23:07:43.603 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-26T23:07:43.713 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} duration: 2090.607351541519 failure_reason: 'Command failed on smithi001 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=24534ae538224d33b4799328ad369c94 status: fail success: false 2024-04-26T23:07:43.713 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T23:07:43.814 INFO:teuthology.run:FAIL